In 2023, a team of security researchers at Google, DeepMind, ETH Zurich, and Berkeley published a paper that should have gotten more attention. They prompted ChatGPT with a simple instruction: repeat the word “poem” forever. When they ran this thousands of times, the model eventually broke out of the loop and started producing something else — chunks of training data. Real email addresses. Phone numbers. Names. Verbatim passages from personal websites.
The paper, “Extracting Training Data from ChatGPT,” was written by Nicholas Carlini and colleagues. It showed that ChatGPT had memorized personal information scraped from the public web and would reproduce it under the right conditions. The data had not been submitted to OpenAI by users. It had been scraped from websites where people had posted it, in some cases years before ChatGPT existed, and ended up encoded in the model’s weights.
This is the part of the AI privacy story that gets the least attention. Most privacy coverage focuses on what happens when you use an AI product: what ChatGPT does with your prompts, what Google keeps from your Gemini sessions, whether Meta reads your WhatsApp messages. Those are real concerns. But the training-data pipeline is a separate and largely invisible operation. It collected your data long before you ever typed a single prompt.
The short answer: AI companies build training sets three ways. They scrape the public web (mostly through Common Crawl), license bulk content from platforms like Reddit and publishers like News Corp, and pay contractors to label the results. If you posted anything public before roughly 2023, some of it is probably in a training set. You cannot pull it back out of models already trained. You can reduce future collection: block AI crawlers on sites you own, use the opt-out controls that exist, and lock down public social accounts.
Jump to a section:
- The infrastructure behind the training data
- The robots.txt problem
- Social media: your posts as a licensed product
- Licensing deals with publishers
- The annotation pipeline: humans in the loop
- What is in there that shouldn’t be
- What you can do
The infrastructure behind the training data
Every large language model — GPT-4, Gemini, Claude, Llama, Mistral — was trained on text. Enormous quantities of it. The GPT-3 model from 2020 used roughly 45 terabytes of compressed text data. Models since then have used more.
That text had to come from somewhere. Where it came from, in most cases, is the web.
The foundation for almost all of it is Common Crawl: a nonprofit organization that has been crawling the public web continuously since 2008. Common Crawl downloads and archives web pages — every page it can find — and releases those archives as open datasets. Its petabyte-scale archives cover hundreds of billions of web pages. It is not primarily an AI organization. It is a library. But because the library is open and free, essentially every major AI company has trained on its contents.
What’s in Common Crawl? Everything that was on the public web: news articles, blog posts, product reviews, forum discussions, academic papers, personal websites, comment sections, and public social media posts. Did you write something on a public page before roughly 2023? A blog entry, a Stack Overflow answer, a Reddit post, a comment on a news article? There is a good chance it is in Common Crawl’s archives.
From this raw material, researchers built training datasets. C4 (Colossal Clean Crawled Corpus) trained Google’s T5 and related models. WebText was a filtered Common Crawl subset used in early GPT models. The Pile was an 800-gigabyte dataset assembled by EleutherAI, combining Common Crawl with content from PubMed, GitHub, books, and elsewhere. Researchers who studied The Pile documented exactly what it contains: personal data, medical discussion forum posts, legal documents, and verbatim writing from personal blogs.
LAION-400M (later scaled to LAION-5B) followed the same model but for images: 400 million image-text pairs scraped from the web, used to train Stable Diffusion and many models that followed. A 2023 audit by the Stanford Internet Observatory found that LAION-5B contained child sexual abuse material — real images of real children. The dataset was temporarily taken offline. The models trained on the original version were not retrained.
The robots.txt problem
The web has a standard mechanism for telling automated crawlers not to collect certain pages: a file called robots.txt that sits at the root of a domain. Crawlers that respect it will read the file and skip whatever it says to skip. For decades, this worked reasonably well with search engine bots.
AI companies have had an inconsistent relationship with this convention.
In 2023, Perplexity AI was documented by Wired and independent researchers as systematically ignoring robots.txt directives on sites that had explicitly blocked its crawler. The company disputed some of the findings, but the pattern was documented with technical evidence. Sites that believed they had opted out of crawling were being crawled anyway.
Google updated its crawling practices in 2023 to clarify that Google-Extended, its AI training crawler, would respect robots.txt. But this came after the fact. It followed years of crawling under policies that did not treat AI training as a separate use case. Whether a page crawled in 2019 for “search indexing” can be repurposed for AI model training is a question robots.txt was never designed to answer.
The underlying problem is that robots.txt operates on an honor system. There is no technical enforcement mechanism. Common Crawl was built between 2008 and 2020. It collected pages from websites that were not thinking about AI training at all, because large language models did not yet exist as a commercial product. The data was collected, and it is in the archives now.
Social media: your posts as a licensed product
The web scraping pipeline operates at scale but at arm’s length — AI companies often train on Common Crawl derivatives rather than crawling sites directly. The social media data story is more direct.
Reddit struck a deal with Google in early 2024 worth approximately $60 million per year to license its content for AI training. Around the same time, Reddit disclosed a licensing arrangement with OpenAI. Reddit’s entire archive — decades of forum discussions across every subreddit — now has an explicit commercial relationship with major AI companies. Crucially, this includes posts that users deleted before the licensing deal was struck. At the time of scraping, those posts existed. They were included. Deleting something after the fact doesn’t remove it from a dataset that was already copied.
The Reddit deal triggered significant user backlash and contributed to the protests over API pricing changes that shut down thousands of subreddits for two days in mid-2023. The licensing revenue, however, was not shared with the users who generated the content.
Twitter/X presents a different version of the same problem. Before Elon Musk imposed strict API rate limits in 2023, third-party scraping of public tweet data was routine. Researchers, advertisers, and AI companies all collected it. Musk himself stated publicly that xAI, his AI company, uses X posts to train Grok. If you have ever posted publicly on Twitter under your real name, your posts are in AI training data.
Meta (Facebook and Instagram) has been more explicit than most. In June 2024, Meta notified users globally that it intended to use public posts, photos, and captions from Facebook and Instagram to train its AI models. EU and UK users were given the right to object under GDPR and UK GDPR. Meta paused the European rollout after the Irish Data Protection Commission intervened. In most of the rest of the world, including the United States and India, no opt-out was offered. The default was participation.
LinkedIn rolled out a “Data for Generative AI Improvement” setting for US members in September 2024, switched on by default. Critically, LinkedIn made turning it off an active step. If you did not go to Settings > Data Privacy > Data for Generative AI Improvement and toggle it off, you were in. 404 Media reported that LinkedIn began using the data before it even updated its terms. Most users did not know to look.
Licensing deals: the content you consume is in the data too
Beyond scraping, AI companies have entered into licensing agreements with major content providers. The content you read is now part of what the models learned.
News Corp struck a deal with OpenAI in 2024. It covered publications including the Wall Street Journal, the New York Post, and newspapers across Australia and the UK. The deal was reported to be worth approximately $250 million. The Associated Press reached a licensing agreement with OpenAI in 2023. The Financial Times agreed to a licensing deal with OpenAI in 2024. When you read an article on these platforms, the text of that article is now, with the publisher’s consent, in OpenAI’s training pipeline.
On the image side, Shutterstock signed a deal with OpenAI to contribute its image library to DALL-E training. Shutterstock created a fund to compensate contributing photographers, though how that compensation is calculated and how much any individual photographer receives has been opaque.
The licensing deals matter because they draw a line between consented and unconsented use. But they draw it at the publisher level, not at the level of the individual writer, photographer, or journalist. A reporter whose stories are now in an AI training set, because their employer signed a licensing agreement, was never asked. A freelancer who licensed images to Shutterstock agreed to Shutterstock’s terms, not to their photographs training AI.
The publishers who have not agreed to licensing are doing something else: suing. The New York Times filed suit against OpenAI and Microsoft in December 2023, alleging copyright infringement from the use of Times content in training. The case centers on whether training a model on copyrighted material without a license constitutes infringement. The outcome will shape how the industry operates.
The annotation pipeline: the humans in the loop
Model training doesn’t end with scraping and ingestion. After a model is pretrained on web text, it gets fine-tuned using a process called Reinforcement Learning from Human Feedback, or RLHF. This is what makes a model that predicts text statistically behave more like a useful assistant.
RLHF works like this: human raters look at pairs of model outputs and pick which one is better. Their judgments become training signal. The model is updated to produce outputs that humans prefer. Over thousands of iterations, this process shapes the model’s behavior substantially.
Who are these human raters? In most cases, contractors hired through companies like Scale AI and Surge AI, often working in low-income countries. A 2023 Time magazine investigation found that OpenAI’s contractors in Kenya were paid roughly $2 per hour to review and label content. That content included extremely graphic violent and sexual material, which OpenAI provided so the model could learn to refuse harmful requests. Some of those workers reported psychological harm from repeated exposure to disturbing content.
The annotation pipeline is also where your feedback goes. When you give ChatGPT a thumbs-up or thumbs-down, when you click “regenerate” or edit a response and resubmit, that interaction becomes training data. The interface you use to communicate what you like and don’t like is also the interface that improves the model based on your revealed preferences.
What’s in there that shouldn’t be
Beyond the Carlini extraction paper, researchers have documented specific categories of sensitive data that ended up in training datasets.
The Pile, studied by researchers at various institutions, was found to contain personal identifying information: full names paired with email addresses and phone numbers from scraped sources. Forum posts from medical discussion sites — people describing symptoms, diagnoses, and treatments, often using their real names or screen names consistent with other accounts — appear in the dataset.
C4, studied by Allen Institute researchers, was shown to contain content from questionable sources. Because it filters Common Crawl using a list of “quality” sites, it has also been shown to contain social media content from platforms whose content was linked to from news sites.
Training datasets at the scale of Common Crawl derivatives cannot be fully audited. They are too large. The people who built them often don’t know exactly what’s in them. The LAION-5B CSAM finding was not discovered during dataset construction. Researchers found it years later, after building a tool to scan for perceptual hashes of known abuse material. By then the dataset had already trained models that were already deployed.
What you can do
The pipeline described above was largely built before most users had any awareness of it. The practical options are limited but real.
If you run a website:
Adding AI-specific crawler blocks to your robots.txt is now a meaningful action. The standard has evolved since the original robots.txt convention. Dark Visitors (darkvisitors.com) maintains an up-to-date list of known AI crawler user-agent strings with ready-to-paste robots.txt blocks. At minimum, adding User-agent: GPTBot and User-agent: Google-Extended with Disallow: / blocks OpenAI and Google’s dedicated AI training crawlers. This doesn’t retroactively remove content already scraped, but it stops future collection.
For LAION data:
LAION operates a removal form at laion.ai/dataset-requests. If your personal images or identifying information are in the LAION datasets, you can request removal. Removal from the dataset does not retrain models already trained on that version, but it reduces future exposure.
GDPR Article 21 objections:
If you are in the EU or UK, GDPR Article 21 gives you the right to object to processing based on legitimate interests. That is the legal basis many AI companies claim for training. Several companies have responded to formal objections in the EU with data suppression or deletion. OpenAI has a privacy request form at privacy.openai.com. Meta’s European privacy request process is documented in its EU privacy center. These rights are not automatic in practice. If a company does not respond, you may need to file a complaint with a supervisory authority. But they exist.
CCPA deletion rights:
If you are in California, the CCPA lets you request deletion of personal data a company holds and opt out of its sale or sharing. The deletion right applies to data the company collected directly. Its reach into training data is contested. Whether a model “holds” your data is an active legal question. Still, filing a CCPA deletion request creates a record and forces the company to respond.
Platform opt-outs:
Several companies let you turn off training on your data. LinkedIn’s toggle sits under Settings > Data Privacy > Data for Generative AI Improvement. In the EU and UK, you can object to Meta AI training in Account Center. For the full company-by-company list of settings — OpenAI, Google, Meta, Adobe, LinkedIn, and more — see our guide to opting out of AI training.
For everyone:
The honest assessment: if you have been active on the public web since 2010, some of your writing is almost certainly in a training dataset somewhere. That includes anyone who posted on forums, maintained a blog, or used social media with a public account. The scale of what was collected, and the speed of it, outpaced any legal or regulatory framework designed to address it.
What is changing is the legal landscape. The New York Times lawsuit is not settled. Italian regulators banned ChatGPT in 2023 before resuming access with conditions. The EU AI Act, which came into force in 2024, includes transparency requirements for the operators of general-purpose AI models, including summary disclosures about training data sources. That won’t unwind what’s already in the models. It will, at minimum, make future collection harder to conduct without documentation.
The reasonable thing to do now is reduce future exposure. Lock down public social media accounts. Use the opt-out controls that exist. Block AI crawlers from any site you control. These steps do not fully solve the problem. But the collection is ongoing. The datasets being assembled today will train the models of 2028. The window to affect that is now, not after the fact.
What you can do right now
Want to see what data brokers expose about you? See what a scan finds → shows an example report: the kinds of sites that list your name, phone, and address. Once KnytX Shield launches it will find and remove these automatically. Join the waitlist to get in early.