The Models Are Training on AI-Generated Content. Nobody Knows What Breaks First.

In 2021, if you scraped the internet for training data, you got human-generated text.

Blog posts written by people with opinions. Forum threads where humans argued with each other. News articles reported by journalists who had been somewhere and seen something. Stack Overflow answers from developers who had debugged the actual problem. Wikipedia edits from people who cared enough about accuracy to correct each other.

In 2026, if you scrape the internet for training data, you get something different.

A significant and growing fraction of what you collect was written by a model. Marketing copy. SEO articles. Product descriptions. LinkedIn posts. GitHub READMEs. Customer support responses. News summaries. Code comments. Tutorial content.

The models are everywhere. The content they produce is everywhere. The training pipelines that collect internet text to train the next generation of models are collecting that content too.

The researchers who study what happens next call it model collapse.

The labs building the next generation of models are not talking about it publicly.

What model collapse actually means

The theory is straightforward.

A model is trained on human text. It learns the statistical patterns of human language. It produces outputs that approximate those patterns.

When those outputs are fed back into the training data for the next model, the next model learns from approximations rather than originals. The subtle errors in the first model’s outputs become part of the training signal for the second. The second model’s approximations are slightly further from human text than the first’s.

The third model trains on the second model’s outputs. Further still.

The researchers who have studied this experimentally describe a specific failure mode. The tails of the distribution collapse first. The rare words, the unusual phrasings, the edge cases of human expression. The model stops generating them because they are underrepresented in the synthetic training data.

Over successive generations, the output becomes more uniform. More central. More average. The diversity that made human language rich and the variety that made it useful for training compresses toward a mean that nobody chose.

The scale that makes this concrete

The percentage of internet text that is AI-generated has grown faster than most estimates predicted.

Content farms that used to employ human writers have replaced them with model pipelines. The economics are overwhelming. A human writer costs money per article. A model costs fractions of a cent.

SEO content, which represents an enormous fraction of the text on the indexed internet, has been predominantly AI-generated for over a year at the scale-oriented end of the market.

Customer support knowledge bases. Product documentation. Translated content. Templated news. Social media posts from accounts optimising for engagement rather than authenticity.

The estimate that gets cited most often in research discussions is that by 2027, the majority of new text published to the internet will have been generated or substantially modified by AI.

The majority.

The training pipelines for models launching in 2027 and 2028 will collect this content. The filters designed to remove low-quality text were calibrated against human-generated low-quality text. AI-generated text that is grammatically clean but semantically hollow passes through many of these filters without triggering them.

The lab that studied it and stopped publishing

Here is what makes this more than theoretical concern.

Multiple research groups have published papers demonstrating model collapse in controlled settings. The effect is real. The direction is consistent. Successive generations of models trained on synthetic data degrade in measurable ways.

The research was published.

The follow-up research, the studies examining what this means at the scale of internet-trained frontier models, has been considerably quieter.

There are a few explanations for this.

The labs that would do this research are the same labs training the frontier models. Publishing research demonstrating that their training approach has a fundamental long-term vulnerability is not in their immediate interest.

The researchers at academic institutions who would study this independently face the problem that internet-scale training data is not publicly available. You cannot study the effect on frontier models without access to frontier training pipelines.

The effect is real at small scale. Whether it is real at frontier scale, and how fast it manifests, is genuinely unknown because the people with the data to study it are the people with the least incentive to publish what they find.

The fingerprint problem

There is a related problem that is more tractable and equally concerning.

AI-generated text has statistical fingerprints.

Specific word distributions. Characteristic sentence structures. Particular ways of transitioning between ideas. The tendency to produce certain types of hedging language. The overuse of specific phrases that appear disproportionately in model outputs.

These fingerprints are subtle enough that humans often cannot detect them reliably. They are not subtle to another model.

When model B trains on content generated by model A, model B learns model A’s fingerprints alongside the content.

This has a specific consequence that researchers describe as homogenisation. The diverse stylistic variation that existed in human writing, shaped by individual voice, cultural background, professional context, geographic location, compresses toward the statistical center of whatever models are generating the bulk of the training content.

The next generation of models sound more like each other. Less like the full range of human expression.

Less like the thing they were built to approximate.

What the data poisoning research showed

Security researchers approached this problem from a different angle.

They were not studying accidental model collapse from organic synthetic data accumulation. They were studying deliberate data poisoning. What happens when an adversary intentionally injects synthetic content into the training corpus to shift model behavior.

The findings were not reassuring.

Relatively small amounts of targeted synthetic content can produce measurable shifts in model outputs. The model learns from what it is trained on. If what it is trained on has been shaped, the model has been shaped.

The internet is not being deliberately poisoned for this purpose at scale. But the organic accumulation of AI content is producing an unintentional version of the same effect.

Nobody chose to shift the training distribution toward AI-generated text. The economic incentives for content production chose it for them.

The result is the same either way.

The detection arms race nobody is winning

The response from training teams has been to build better AI content detectors.

Identify AI-generated content. Filter it from training data. Maintain the human signal.

The problem is that AI content detectors work by identifying the statistical fingerprints of AI-generated text. As the models improve, the fingerprints become subtler. As the detectors improve, the models producing content for distribution are optimised to evade them.

It is an arms race between generation and detection.

The generation side has a structural advantage. The models generating content are the frontier models. The detectors are trying to identify the outputs of frontier models. Identifying something requires being at least as capable as the thing you are identifying.

In practice, the frontier models are ahead.

The content being generated by the best models today is systematically harder to detect than the content generated by models from eighteen months ago. The detection rate has not kept pace.

The training pipelines that depend on detection to maintain data quality are running on filters that were effective against yesterday’s generation capability.

What this means for the next three years

The labs are not ignoring this. They are pursuing several responses simultaneously.

Curated human data partnerships. Agreements with publishers, news organisations, and content platforms to access known-human text. The economics of these deals are expensive. The supply is finite.

Synthetic data that is generated intentionally and labeled clearly. Training on AI-generated data is not inherently problematic if it is used deliberately for specific purposes rather than accumulating accidentally in web crawls.

Better provenance tracking. Efforts to make the origin of content verifiable so training pipelines can filter on it. This requires infrastructure that does not broadly exist yet.

None of these is a complete solution. Each helps at the margins.

The fundamental dynamic, that AI content generation is economically dominant for large categories of internet text, and that training pipelines cannot perfectly distinguish it from human content, is not resolved by any of these responses.

The researchers who study this are not predicting imminent collapse. They are predicting a slow degradation in training data quality that will be difficult to attribute cleanly to any single cause and difficult to reverse once established.

Slow degradation is the hardest category of problem to respond to. It does not produce the incident that forces action. It produces the gradual worsening that everyone notices and nobody prioritises.

The thing the benchmarks will hide

Here is the specific reason this matters more than it might seem.

Benchmark performance will not show model collapse clearly.

The benchmarks are fixed. They were designed by humans and have not changed substantially. A model trained on progressively more synthetic data will still perform well on benchmarks designed to test the capabilities that synthetic data transmits.

What benchmarks will not show is the loss in diversity, nuance, and edge-case coverage that human-generated text provided.

The model will score well on the benchmark. It will be subtly worse on the tasks that required the full richness of human linguistic variation. The degradation will live in the space between the benchmark and the real use case.

Exactly where most AI degradation currently lives.

The labs will publish benchmark improvements. The models will be somewhat worse at things the benchmarks do not measure. The users who notice will file support tickets attributed to prompt issues. The root cause will not be investigated because the benchmark says the model is fine.

The signal is already getting noisier.

The question is whether the people responsible for monitoring it are looking in the right place.

Most of them are looking at the benchmark.

The benchmark is not where the problem will appear.