The Hallucination Problem Is Getting Worse. The Industry Gave It a New Name Instead.

In 2023 the problem was called hallucination.

The models made things up. Sometimes obviously. Sometimes convincingly. The word hallucination spread because it captured something real about the failure mode. The model was perceiving something that was not there and reporting it with confidence.

The AI industry did not like the word.

Hallucination implies the system is broken. It implies a medical metaphor that suggests fundamental unreliability. It scared enterprise buyers. It gave ammunition to critics.

So the industry moved to confabulation.

Confabulation is a clinical term from neuroscience. It describes when brain-damaged patients produce false memories without intending to deceive. It sounds more technical. More clinical. More like something that can be managed rather than something that reveals a fundamental flaw.

The word changed. The problem did not.

And the problem is getting worse in a specific way that the language change is making harder to see.

What actually changed with the newer models

The older models hallucinated loudly.

They invented citations that were obviously wrong. They produced statistics that were implausible. They made factual claims about public figures that were easy to check and clearly false.

The errors were real but they were often catchable. A careful reader with domain knowledge could identify them. The confidence of the output sometimes did not match the implausibility of the claim. The tells were there.

The newer models hallucinate quietly.

The errors are more subtle. The invented citations are more plausible. The statistics are in the right order of magnitude. The factual claims are close enough to truth that checking them requires more than casual familiarity with the subject.

The models got better at being wrong in ways that look right.

This is not a minor improvement in the wrong direction. It is a significant shift in the risk profile of deploying these systems in any context where accuracy matters.

A system that is wrong obviously can be caught. A system that is wrong convincingly is a different problem entirely.

The confidence that does not correlate with accuracy

Here is the finding that should be more central to every conversation about AI reliability.

Model confidence does not reliably predict model accuracy.

When a model produces an answer with high confidence, that confidence is not a signal that the answer is correct. It is a signal that the model’s training led it to produce this answer with high confidence. Those are different things.

The calibration research on frontier models shows that on questions outside their reliable knowledge base, models express high confidence at rates that significantly exceed their actual accuracy. They do not know what they do not know. Or more precisely, they do not signal what they do not know.

This matters enormously for production systems.

A team that built an AI system trusts the outputs more when the model sounds certain. The model sounds certain most of the time. The system receives high-confidence wrong answers regularly and has no mechanism to distinguish them from high-confidence right answers.

The confidence is noise. Most teams are treating it as signal.

The domain expert who stopped checking

Here is the failure mode that is costing the most right now.

A team deploys an AI system to assist domain experts. Lawyers reviewing contracts. Doctors reviewing literature. Engineers reviewing specifications. The AI is positioned as a tool to help the expert work faster, not to replace their judgment.

In the first weeks, the experts check the AI outputs carefully. They catch errors. They correct them. They maintain appropriate skepticism.

Over time the skepticism fades.

The outputs have been mostly right. The checking has been tedious and has almost always confirmed what the AI said. The expert is busy. The AI has built up a track record that justifies, in the expert’s mind, reducing the verification overhead.

Then the AI produces a confident, plausible, wrong answer in a high-stakes situation.

The expert, no longer checking carefully, misses it.

The failure is not the AI’s hallucination. The AI hallucinated from the beginning. The failure is the erosion of the human oversight that was supposed to catch it.

The AI’s track record of being mostly right created the conditions for the costly failure when it was confidently wrong.

The enterprise sale that did not mention this

Every major AI vendor has a slide about reliability.

The slide discusses accuracy improvements between model versions. It shows benchmark performance. It highlights the systems in place to reduce fabrication. It describes the evaluation frameworks that measure output quality.

The slide does not discuss how confidence calibration works or does not work. It does not discuss the conditions under which the model is most likely to be convincingly wrong. It does not discuss the failure mode where user trust in the system outpaces the system’s actual reliability.

The enterprise buyer leaves the meeting with an accurate understanding of the model’s average performance on benchmarks.

They leave with no understanding of the tail risk. The cases where the model is wrong in ways that will not be caught. The workflows where the human oversight that is supposed to catch errors has eroded. The high-stakes decisions where a plausible confident wrong answer gets acted on.

The vendors are not lying. They are selecting what to disclose.

The gap between what is disclosed and what matters is where the enterprise deployments that will generate bad headlines in the next twelve months are being built right now.

The three applications most at risk

Not all AI applications carry the same hallucination risk.

The ones with the lowest risk are the ones where errors are immediately visible and inconsequential. AI-generated first drafts that a human rewrites substantially. AI suggestions in creative contexts where the human makes all final decisions. AI-assisted brainstorming where the output is a starting point not a conclusion.

The ones with the highest risk are the ones where three conditions coincide.

The outputs are acted on directly with minimal human review. The domain is specialised enough that errors require expertise to detect. The model sounds confident regardless of accuracy.

Legal document review at scale. Medical literature synthesis for clinical decisions. Financial analysis informing trades. Compliance checking against regulations the model may have learned incompletely.

In each of these cases, the combination of reduced human oversight, specialised domain, and overconfident wrong outputs creates a failure mode that does not announce itself until it has already caused damage.

The teams deploying AI in these categories have mostly implemented guardrails appropriate for the average case.

The average case is not where they will fail.

What the research actually shows

The academic research on AI reliability over the past eighteen months has produced findings that are not well-represented in the vendor conversations.

Newer models produce fewer obviously false statements but more subtly false statements. The rate of detectably wrong outputs has fallen. The rate of undetectably wrong outputs has not fallen at the same rate, and on some measures has increased.

The models have gotten better at sounding right. They have not gotten proportionately better at being right.

The gap between sounding right and being right is the space where the next generation of AI failures will occur.

The teams that understand this are building verification layers that do not rely on output confidence as a signal. They are checking outputs against ground truth sources. They are building workflows where AI-generated content that will be acted on passes through domain expert review regardless of how confident the output sounds.

This is more expensive than trusting the model.

It is less expensive than acting on confident wrong answers in high-stakes contexts.

The name that obscured the problem

Hallucination was a bad word for a real phenomenon.

Confabulation is a clinical word for the same real phenomenon.

Neither name helped teams understand the specific risk they were managing. Both invited the assumption that the problem was being addressed because it was being named.

The problem that needs a name is not what happens when the model makes things up.

It is what happens when the model makes things up convincingly enough that the human oversight designed to catch it does not.

That problem does not have a good name yet.

It does not need one to be real.

It needs teams to understand that the confidence the model expresses is not a reliability signal. That newer models being wrong less obviously does not mean they are wrong less often on the questions that matter. That the human oversight designed to catch errors needs to be maintained even after the model has built up a track record that makes maintaining it feel unnecessary.

The model is not hallucinating less.

It is hallucinating better.

Those are not the same thing.

And the teams that have not noticed the difference are the ones the next wave of AI reliability failures will come from.