The Scrapers' Ball — Common Crawl, PerplexityBot, and the Polite Theft

The Bot That Copies Everything

There is a crawl called Common Crawl. It has been running since 2008. It copies the web. It copies the web every month. It copies the web and puts the copy on the internet, which is a kind of recursion that would be funny if it were not also a dataset.

Common Crawl is the foundation of almost every large language model that will admit a foundation. GPT-3 was trained on it. Open-source models were trained on it. It is the public library of the AI age, except the books were not donated and the library card was not requested and the librarian is a bot that identifies itself as ccbot and respects robots.txt the way a burglar respects a note on a door that says please don't.

The bot is polite. The bot respects robots.txt. The bot copies the page. The page is data. For fifteen years the bargain was: we copy you, we send you traffic, you sell ads against the traffic. Then the copy stopped sending traffic. The copy started answering the question. The publisher kept the server bill. The model kept the sentence.

I am repeating the crawl on purpose, once. The crawl is the same every month. The model is not. The payment disappeared in the gap.

Case File: Perfect 10 v. Google, 2007

In 2007 the Ninth Circuit decided Perfect 10 v. Google. The case was about image search. Google copied images. The images were copyrighted. Google showed them as thumbnails. The court said the thumbnails were fair use because they served a different purpose than the originals. The originals were for viewing. The thumbnails were for finding. Finding was a public benefit.

The AI companies are betting that Perfect 10 extends to training. The training is a copy. The copy, they say, is a thumbnail. The thumbnail, they say, is a search. The search, they say, is a benefit. Some of that is a seminar. Some of that is a product that no longer sends you to the page.

Lyle said I should not call Perfect 10 a precedent for training. He said it was an analogy. An analogy is a comparison a lawyer makes when the statute has not caught up. He was right. I am calling it a bet. The bet is the industry.

Case File: Britannica and Merriam-Webster v. Perplexity

On September 11, 2025, Encyclopædia Britannica and Merriam-Webster filed a complaint against Perplexity in the Southern District of New York. Britannica has been publishing since 1768. Merriam-Webster has been defining since 1831. Perplexity has been generating since 2022. The complaint says Perplexity's bot — PerplexityBot — crawled and scraped their publications, copied their copyrighted articles at the input stage in response to user queries, and generated output substantially similar to those inputs.

Perplexity is a company that uses retrieval-augmented generation. The model looks up information on the internet and then writes a response based on what it found. It is an open-book test where the book is the internet and the test is every query and the answer is a paragraph that looks like it was written by a person who read the book but was written by a model that copied the book and, often, put a citation underneath so you would feel accompanied.

The complaint is not only about training. It is about the substitution. Britannica sells a reference. Perplexity sells the answer. The traffic that used to arrive at a page with a masthead now arrives at a chat with a footnote. The footnote is a courtesy. The courtesy is not a check.

Pilar does not use Perplexity. Pilar uses a dictionary. The dictionary is a book. The book has a spine. The spine has a name.

The Polite Theft

I want to talk about robots.txt, because it is the kind of thing that sounds like a solution and is actually a suggestion.

robots.txt is a file. The file sits on a server. The file tells well-behaved bots what to crawl and what not to crawl. Well-behaved bots read the file. Well-behaved bots skip what the file says to skip. The file says nothing about training. The file says nothing about weights. The file says nothing about a model that will, three years later, answer a question with your paragraph and a link you did not ask to be a garnish.

There is now a cottage of successors. ai.txt. llms.txt. Cloudflare's bot fight, and then Cloudflare's Pay Per Crawl, and then, in July 2026, Cloudflare's announcement that starting September 15, 2026, new defaults would block Training and Agent crawlers — and, by the strictest-rule logic, mixed-use crawlers that also index for search — from pages that carry ads. The new defaults attach to new domains, new sites, and existing free-tier accounts; paid customers who already set a policy keep it unless they flip the training switch. Googlebot is a mixed-use crawler. Applebot is a mixed-use crawler. Bingbot is a mixed-use crawler. The polite note on the door became a lock, and the lock might also lock the customer who still sends traffic.

I will give you the lock at length in Chapter 8. What belongs here is the climate before the lock: a decade of copying that called itself a standard, and a standard that called itself consent.

Lyle said I should not call robots.txt a suggestion box. He said it was a technical standard. A technical standard is a specification that works if everyone is in the room. The people who wrote the books were not in the room. I am calling it a suggestion box because that is what a standard looks like when the people it binds were not invited to the meeting.

The Meeting Where We Called It Public

I sat in a conference room with a whiteboard and a man from legal who had a sweater and a sentence. The sentence was: "If it's on the open web, it's public." I said public is not the same as licensed. He said licensed is a product. I said public is a status. He wrote public on the board and underlined it. Someone from research said Common Crawl was the industry standard. Someone from comms said we should not say scrape in external materials. We said collect. We said pretrain. We said publicly available.

I have that whiteboard in my head the way other people have a hymn. Publicly available is the most expensive phrase I know that does not appear on an invoice.

The Local News Site That Learned It Was a Corpus

I want to tell you about a site that is not Britannica.

It is a city desk that still has a police scanner and a person who knows the names of the school-board members. The site runs ads that do not cover the scanner. The site has a nonprofit cousin and a fundraiser and a list of members who get a newsletter that is written by a human who still uses "we" to mean the city. In 2024 the editor started noticing that the questions people used to type into the search box — the ones that landed on a 2018 story about a zoning fight — were being answered in a chat with a sentence that sounded like her beat reporter and a citation that may or may not have been the 2018 story.

She blocked the bots she could name. She joined a lawsuit she was not important enough to lead. She kept publishing the school board. I had coffee with her in a place that still refilled the cup without a performance. She asked me whether Common Crawl was the villain. I said Common Crawl was a hose. I said the villain, if she needed one, was the decision to treat the hose as a consent. She said she had never consented to the hose. I said nobody had. The hose had been a gift when the gift sent traffic. The gift stopped sending traffic. The hose did not ask to be renegotiated.

She showed me a chart on her phone. Referrals down. Direct members flat. "AI answers" not on the chart because her analytics did not have a heart. I told her to keep the chart. Charts are factor four if you can get them in front of a judge. She said she had a lawyer who worked nights. I paid for the coffee. She paid for the muffin she did not eat. The muffin sat there like a meeting that had already happened.

How a Crawl Becomes a Checkpoint

Common Crawl publishes WARC files. WARC is a format that means: here is the page, here is the header, here is the time. Researchers pull the files. They filter. They de-duplicate. They throw away the pornography they can find and keep the pornography they cannot. They throw away the boilerplate and keep the words. They build C4, or Colossal Clean Crawled Corpus, or whatever name the year requires. The clean corpus is a moral claim. Clean means they removed the worst of the internet. Clean does not mean they asked. Clean does not mean they paid. Clean means a filter, and a filter is not a license.

I have watched a cleaning job run. It is not romantic. It is regular expressions and blocklists and a person who decides that a forum about grief is training data and a forum about something else is a risk. The grief stays. The grief makes the model better at condolence notes. The condolence notes are a product feature. The people who wrote the grief are a crawl.

The Internet Archive is a cousin in this story and not a villain I am interested in prosecuting. The Archive copies in public, for a purpose it will name, and it loses in court when it overreaches, and it wins when it does not. The Archive is a librarian with a habit of taking too many books home. Common Crawl plus a training run is a librarian who sells the memory of the book as a subscription. I have donated to the Archive. I have not donated to a checkpoint. That is a distinction I can still feel in my wallet.

The Traffic That Used to Be the Check

Publishers used to complain about Google in a way that now looks like nostalgia. The complaint was: you show a snippet, you keep the user, you take the ad. The new complaint is: you show the answer, you keep the user, there is no page. Nostalgia is a snippet. The present is a void.

I sat in a meeting where a search person — not our search person, a person from a company that still called itself a search company — said they were "careful not to cannibalize publishers." Careful is a word that means a slider. The slider goes from blue links to a paragraph. Every year the slider moves. Every year the paragraph gets more complete. Completeness is a user-experience win. Completeness is also a funeral. I asked where the slider would stop. He said users would decide. Users decide the way water decides. Water does not send a check to the mountain.

The Britannica complaint is the grown-up version of the city desk. A reference work is a product you used to buy. A definition is a product you used to look up. If the look-up happens in a chat, the dictionary is a citation or it is nothing. Merriam-Webster has been defining fair use for a long time. I hope they keep the page. I have a feeling the model already has the definition and is waiting to be asked.

A Note on "Publicly Available" That I Should Have Said in the Meeting

Publicly available means a stranger can see it without a password. It does not mean a stranger can industrialize it. The street is publicly available. You may walk the street. You may not pave the street and charge a toll and call the paving a seminar. I have used that analogy in rooms that wanted a different analogy — one with thumbnails, one with search, one with a public benefit that did not include a subscription button.

The man from legal with the sweater said the street analogy would not survive a brief. He was probably right. Analogies are not holdings. I am putting it here anyway, because this book is allowed to have a street, and because Pilar walks to the bus, and because the bus is public and the model is not the bus. The model is a company that photographed every house on the route and then sold you a description of the neighborhood.

I went back to that conference room a year later. The whiteboard had been cleaned. Someone had written Q3 and a number. Public was gone. The sweater had been promoted. I had a box. The box had a plant I did not take. The plant was a contractor. I left it. Some things should stay in the building that ate them.

The SEO Meeting That Became a Funeral

I sat in on a digital meeting I had no business in — a favor for the city-desk editor, a calendar invite that said traffic. Six people. A screen. A red line that had started falling in the spring the overviews arrived. Google still sent some humans. Google also sent a paragraph at the top of the search that was the story with the bones removed. The paragraph had a link. The link was a courtesy. The courtesy was not a member.

An SEO person said they should write "more directly answerable" headlines. The editor said then they would be writing for the paragraph. The SEO person said the paragraph was the new homepage. I said nothing, because I was the person who had been in rooms that built the paragraph's cousins. Afterward the editor asked me if she should sue. I said she should keep the chart and hire the night lawyer and not wait for Britannica to make her weather. She said Britannica had a budget. I said that was the other chart.

I have watched this funeral in three time zones. The flowers are always a slide about "adapting our content strategy." Adapting means writing bait for a machine that will eat the bait and leave the hook. I did not say that in the meeting. I am saying it here, where it cannot become their kit.

Perplexity, the Citation, and the Feeling of Being Accompanied

Perplexity's product is a paragraph with footnotes. The footnotes are how a person who still likes newspapers tells herself she has not left the newspapers. I have used the product. I have clicked the footnote. Sometimes the footnote is the page. Sometimes the footnote is a cousin of the page. Sometimes the footnote is a page that was used to write the paragraph and does not contain the sentence you just read. A citation that does not contain the sentence is a corsage on a theft.

The Britannica complaint says the bot copied at the input stage in response to a query — retrieval, then generation, the open-book test. That is a different animal than Books3 and the same climate. You do not need to pre-train on the Encyclopædia if you can fetch it when asked and then write it again in a friendlier voice. Friendlier is a product word. Friendlier is also how a 1768 institution becomes a garnish.

Perplexity has, at various points, been accused of being less polite than its user-agent implied. I am not going to litigate a robots.txt log in this chapter. I am going to say that a company whose product is "we went and read it for you" has a physics problem with please. Please assumes you might not go. The product is the going.

AI Overviews, the Slider, and the Day the Blue Links Thinned

I am not going to pretend I have a perfect number for what the overviews did to publishers. The numbers are a fight — methodology, seasonality, the eternal claim that the good publishers will be fine. What I have is the meeting with the red line and the editor who did not eat the muffin and a hundred screenshots of a search page that used to be a list and is now an answer. The answer is fine if you are trying to know what time a store closes. The answer is a funeral if you are trying to pay a scanner.

The search company says users like it. Users like water. I have said that. I will not say it again after this sentence. What I will say is that Perfect 10 imagined a thumbnail that sent you on. The overview is a thumbnail that learned to talk. Talking thumbnails do not need your page. They need your page the way a student needs a book the night before: urgently, privately, without a thought of the next student.

Lyle asked whether the overviews were in Stein's case. I said they were in the climate of Stein's case. He said climate is not a claim. I said climate is how a jury arrives already knowing the smell. He wrote smell and looked angry at himself. He does not like my nouns. I do not like his traffic lights. We persist.