For three decades, a tiny text file has kept the internet from chaos. This text file has no particular legal or technical authority, and it’s not even particularly complicated. It represents a handshake deal between some of the earliest pioneers of the internet to respect each other’s wishes and build the internet in a way that benefitted everybody. It’s a mini constitution for the internet, written in code.
It’s called robots.txt and is usually located at yourwebsite.com/robots.txt. That file allows anyone who runs a website — big or small, cooking blog or multinational corporation — to tell the web who’s allowed in and who isn’t. Which search engines can index your site? What archival projects can grab a version of your page and save it? Can competitors keep tabs on your pages for their own files? You get to decide and declare that to the web.
It’s not a perfect system, but it works. Used to, anyway. For decades, the main focus of robots.txt was on search engines; you’d let them scrape your site and in exchange, they’d promise to send people back to you. Now AI has changed the equation: companies around the web are using your site and its data to build massive sets of training data, in order to build models and products that may not acknowledge your existence at all.
The robots.txt file governs a give and take; AI feels to many like all take and no give. But there’s now so much money in AI, and the technological state of the art is changing so fast that many site owners can’t keep up. And the fundamental agreement behind robots.txt, and the web as a whole — which for so long amounted to “everybody just be cool” — may not be able to keep up either.
In the early days of the internet, robots went by many names: spiders, crawlers, worms, WebAnts, web crawlers. Most of the time, they were built with good intentions. Usually, it was a developer trying to build a directory of cool new websites, make sure their own site was working properly, or build a research database — this was 1993 or so, long before search engines were everywhere and in the days when you could fit most of the internet on your computer’s hard drive.
The only real problem then was the traffic: accessing the internet was slow and expensive, both for the person seeing a website and the one hosting it. If you hosted your website on your computer, as many people did, or on hastily constructed server software run through your home internet connection, all it took was a few robots overzealously downloading your pages for things to break and the phone bill to spike.
Over the course of a few months in 1994, a software engineer and developer named Martijn Koster, along with a group of other web administrators and developers, came up with a solution they called the Robots Exclusion Protocol. The proposal was straightforward enough: it asked web developers to add a plain-text file to their domain specifying which robots were not allowed to scour their site, or listing pages that are off limits to all robots. (Again, this was a time when you could maintain a list of every single robot in existence — Koster and a few others helpfully did just that.) For robot makers, the deal was even simpler: respect the wishes of the text file.
“Cabrini” is a biographical drama directed by Alejandro Gómez Monteverde and written by Rod Barr, based on a story by both. The film depicts the life of Catholic missionary Francesca Cabrini, as she encounters resistance to her charity and business efforts in New York City. The trailer piqued my curiosity, compelling me to watch the […]
In the spring of 2023, after her third case of COVID-19, Jennifer Robertson started to feel strange. Her heart raced all day long and she could barely sleep at night. She had dizzy spells. She felt pins and needles in her arm, she says, a “buzzing feeling” in her foot, and pain in her legs and lymph nodes. She broke out in a rash. She smelled “phantom” cigarette smoke, even when none was in the air.
Robertson, 48, had a feeling COVID-19 might have somehow been the trigger. She knew about Long COVID, the name for chronic symptoms following an infection, because her 11-year-old son has it. But “he didn’t have anything like this,” she says. “His set of symptoms are totally different,” involving spiking fevers and vocal and motor tics. Her own experience was so different from her son’s, it was hard to believe the same condition could be to blame. “I just thought, ‘It’s really coincidental that I never got well, and now I’m getting worse,’” she says.
She saw a doctor in Cyprus, where her family was living at the time, and then in Saudi Arabia, where her husband was working. Neither visit yielded much. Then, after Robertson’s family moved to Scotland in the summer of 2023, a specialist there diagnosed with Long COVID. She is still sick—and a reinfection late last year set her back—but she has found some relief in treatments prescribed by her doctors, including heart medication and antihistamines.
Robertson’s story highlights the many challenges of detecting, diagnosing, and treating Long COVID. It affects people from all walks of life and produces a vast array of symptoms that can range in severity from mild to life-altering. And because there are so many forms Long COVID can take, it can be difficult for patients and doctors to know what’s going on.
That means many people aren’t getting diagnosed or treated, says Nisreen Alwan, a professor of public health at the U.K.’s University of Southampton who studies Long COVID (and has had the condition herself). Alwan’s research suggests there is “considerable self-doubt” among Long COVID patients, with many people questioning if they should get medical care or have the condition at all. That may be in part because media coverage tends to showcase a specific type of patient—someone who is very sick, potentially to the point of being bed-bound, and battling extreme fatigue and brain fog—so people with milder or more unusual symptoms aren’t sure whether their illness counts as Long COVID, Alwan says.
It’s hard to blame people for being confused. Long COVID is so broadly defined that virtually any unexplained health issue that comes after a case of COVID-19 and lasts at least a couple months could fit the bill. More than 200 symptoms have been linked to Long COVID—everything from insomnia and hallucinations to tremors and gastrointestinal issues—and they often look very different from those of an acute COVID-19 case. Further complicating matters, some people feel better for weeks or months after their initial infection before their health deteriorates.
Everything you do—walking to your yoga class, making your favorite latte order, talking to your bestie, and just getting through the workday—happens thanks to your brain. Your brain is the control center for your entire body—it’s how you get shit done. So how can you take care of such a beautifully complex and integral part of your body and keep it in great shape for as long as possible?
Lara V. Marcuse, MD, a board-certified neurologist and codirector of the Mount Sinai Epilepsy Program at the Icahn School of Medicine at Mount Sinai, shares the one thing she does every day (or almost every day, because life gets busy, folks!) to keep her brain healthy. As a bonus? It’s fun.
Pick up a difficult new skill, even if you suck at it.
“I started playing piano in my mid-40s,” Dr. Marcuse tells SELF. It all started by chance when her son began taking lessons: “I took his lesson book on the sly one night before bed, and I was totally enthralled by it,” she says, though she admits she found the songs themselves hard to get into at first. “I’m a 1980s New York City club kid. I grew up on a steady diet of house music, and I never liked classical.” It’s been seven years since she first gave playing a Chopin piece a shot, and she hasn’t looked back since. “[Playing piano] helps me get into [the] nooks and crannies of myself—and into my spirit,” she says.
Taking up a hobby that’s unfamiliar and even difficult forces your brain to exercise new or rarely used neural pathways, and that can help prevent cognitive decline and even protect your brain against Alzheimer’s disease, a type of dementia that leads to memory loss and an inability to complete daily tasks. Keeping your brain active makes neural pathways strong—and the opposite is true if you’re not finding ways to engage your mind.
by Hazel Straub Father God, help me! I fully surrender and give you my heart, my will, for your will, and to let go of pride, stubbornness, resentment, offense, unbelief, and everything I substituted for a relationship with you, and humbly ask for your help. Father God, holy is your name. Change my heart, O […]
Scientists have figured out a way to mass-produce solar panels made out of a so-called miracle material that can massively boost their efficiency.
Perovskite has been hailed for its potential to revolutionize renewable energy, however, converting its record-breaking success in the lab into commercial solar panels has proved difficult due to durability and reliability issues.
A major study into possible production methods for the technology has now concluded that a vacuum-based approach could allow the next-generation solar panels to be manufactured on a commercial scale.
A team led by the Karlsruhe Institute of Technology in Germany and the National Renewable Energy Laboratory (NREL) in the US found that vacuum processes – used to make everything from smartphones to LEDs – held significant advantages over the solvent-based approach usually used to make lab-scale solar cells.
“Vacuum-based processes have proven themselves in the industry for many decades,” said Ulrich W. Paetzold, a professor at the Institute of Microstructure Technology and Light Technology Institute at KIT.
“Although they can decisively advance the commercialization of solar cells, they are heavily underrepresented.”
Solar cells that use a combination of perovskite and silicon have demonstrated vastly more potential to generate electricity from the Sun’s energy compared to traditional silicon cells.
Elon Musk made some striking predictions about the impact of artificial intelligence (AI) on jobs and income at the inaugural AI Safety Summit in the U.K. in November.
The serial entrepreneur and CEO painted a utopian vision where AI renders traditional employment obsolete but provides an “age of abundance” through a system of “universal high income.”
“It’s hard to say exactly what that moment is, but there will come a point where no job is needed,” Musk told U.K. Prime Minister Rishi Sunak. “You can have a job if you want to have a job or sort of personal satisfaction, but the AI will be able to do everything.”
While admitting this prospect may not make “people comfortable,” Musk seemed optimistic about what he termed a “protopian” AI-driven future.
“I think everyone will have access to this magic genie, and you’re able to ask any question. It’ll be certainly bigger for education. It’ll be the best tutor,” he said. “And there will be no shortage of goods and services. It will be an age of abundance.”
Musk’s concept of “universal high income” appears to be an evolution of the universal basic income (UBI) idea supported by other tech leaders like Sam Altman.
“We won’t have universal basic income. We’ll have a universal high income,” Musk said, though he did not explicitly define the difference. “In some sense, it’ll be somewhat of a leveler, an equalizer.” It’s worth noting this isn’t the first time Musk addressed the topic. In 2018, he posted on X: “Universal income will be necessary over time if AI takes over most human jobs.”
Film and Writing Festival for Comedy. Showcasing best of comedy short films at the FEEDBACK Film Festival. Plus, showcasing best of comedy novels, short stories, poems, screenplays (TV, short, feature) at the festival performed by professional actors.
:format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/25282612/246992_AI_at_Work_TXT_FILE_ECarter.png)
Illustration by Erik Carter
