Everyone Knows AI Needs Proper Evaluation. Nobody Is Actually Doing It.

There is a gap in AI development that nobody admits to openly but almost everyone is living inside.

Every practitioner knows that proper evaluation is the difference between an AI system you can trust and one you are hoping works. Every conference talk mentions it. Every best practices guide leads with it. Every post-mortem from a failed AI deployment cites its absence as a contributing factor.

And yet the number of AI systems in production today with rigorous, maintained, task-specific evaluation frameworks is a small fraction of the total.

The rest are running on vibes and inshallah.

Not because the teams are careless. Because building real evaluation is genuinely hard, genuinely expensive, and genuinely unglamorous in a way that makes it consistently lose to shipping features when someone has to make a call about where the next sprint goes.

The feature ships. The evaluation does not get built. The cycle repeats. The technical debt accumulates in the one place that nobody can see until the system fails in a way that a proper eval would have caught.

What evaluation actually requires

The reason most teams skip real evaluation is that real evaluation is not one thing. It is four things, each requiring different work, and all four are necessary for the framework to mean anything.

The first is a representative dataset. Not a collection of examples that make the model look good. A systematic sample of the actual inputs the model will encounter in production, including the edge cases, the adversarial inputs, and the cases where the model is most likely to fail. Building this requires understanding your actual user distribution, which most teams have never formally studied.

The second is ground truth labels. For every example in your eval set, you need a known correct answer. Producing ground truth requires domain expertise, careful annotation, and a process for resolving disagreement between annotators. It is expensive, time-consuming, and cannot be automated without defeating the purpose.

The third is metrics that actually measure what matters. Not benchmark scores. Not BLEU or ROUGE or whatever the paper used. Metrics tied to the specific downstream outcome your AI system is supposed to produce. For most business AI systems, the right metric has never been formally defined.

The fourth is a process for running evaluation continuously and acting on the results. A eval set that gets run once at launch and then sits dormant is not an evaluation framework. It is a one-time audit.

Most teams have partial versions of one or two of these. Very few have all four.

The benchmark that replaced the eval

Here is what most teams actually do instead of building proper evaluation.

They run the model against a public benchmark. MMLU. HumanEval. MT-Bench. One of the dozens of standardised test sets that the research community has produced.

The benchmark score goes up with each model upgrade. The team reports improvement. The system ships.

The benchmark was not designed for their use case. It measures general capability on tasks the research community cared about when they designed it. The tasks your AI system performs in production may have almost no overlap with what the benchmark tests.

A coding assistant that scores well on HumanEval but fails consistently on the specific frameworks and patterns your team uses is a coding assistant your team cannot trust. The benchmark said it was good. The benchmark was not measuring the right thing.

This substitution happens constantly. Benchmark performance becomes a proxy for production performance. The proxy is accepted because the real measurement is hard and the proxy number is available.

The gap between the proxy and the reality lives in production, where it manifests as the AI being worse than expected in ways that are hard to explain to stakeholders who saw the benchmark numbers.

The eval that worked for the demo

There is a specific version of this that deserves its own section.

Many teams do build an eval set. They build it during development, while they are iterating on the system. They pick examples that represent the use cases they are thinking about. They tune the system against those examples. The system performs well on the eval set.

Then they ship.

The eval set was built from the cases the developers imagined. Production is full of cases the developers did not imagine.

The eval passed because the system was tuned against it. It was not tuned against production reality because the team did not have production data when the eval was built.

This is the evaluation equivalent of overfitting. The system is good at the test. The test does not represent the problem.

The teams that avoid this use holdout sets they never touch during development. They collect production data from beta users before wide launch. They update the eval set regularly with real examples from real users encountering real problems.

Each of these practices requires discipline that is hard to maintain under shipping pressure.

The model upgrade that broke things silently

Here is the incident that makes teams regret not having proper evaluation.

A new model version is available. The benchmarks are better. The team upgrades. The system is tested informally. Nothing obviously breaks. The upgrade ships.

Three weeks later, customer support starts seeing a new category of complaint. The AI is handling a specific type of request differently than before. Not catastrophically. Just differently. In a way that is wrong for the use case.

The team investigates. The new model version behaves differently on the edge cases that matter most for this specific application. The benchmark did not test those cases. The informal testing did not encounter them. The production traffic found them immediately.

With a proper eval set, the degradation would have appeared in the evaluation run before the upgrade shipped. The team would have seen it, investigated, and either fixed the prompting or delayed the upgrade.

Without one, the degradation appeared in the customer complaints.

The cost of the degradation, the support tickets, the customer trust, the engineer time to investigate, is almost always higher than the cost of building the eval set that would have caught it.

The teams that have lived through this build the eval set next. The teams that have not yet lived through it will discover it eventually.

What the org chart does to evaluation

There is an organisational reason that evaluation stays underdeveloped that is worth naming directly.

Evaluation benefits are distributed across the organisation. Fewer production incidents. Higher reliability. Better customer outcomes. Faster, safer model upgrades.

Evaluation costs are concentrated on the team that builds it. Engineer time. Annotation budget. The ongoing cost of maintaining the dataset and running the framework continuously.

The team that would build the evaluation often cannot directly capture the benefit. The benefit shows up as an absence of incidents, which is invisible. The cost shows up as reduced velocity on features, which is very visible.

Every quarter, the decision about whether to build proper evaluation is made implicitly by choosing to ship features instead. Nobody makes the explicit decision not to evaluate. They just make the explicit decision to ship the feature, and the evaluation does not happen.

This continues until the incident that makes the cost of not evaluating concrete and attributable.

The org chart that would fix this is one where the team responsible for AI reliability has budget for evaluation infrastructure and is measured on production AI quality rather than feature velocity. Most org charts are not this one.

The teams that actually do it

The AI teams with proper evaluation frameworks share one thing.

They were forced to build them.

Either by a significant production failure that made the cost of not evaluating undeniable. Or by a regulatory requirement that mandated it. Or by a customer contract that required demonstrated accuracy before deployment. Or by a technical leader who had lived through the failure at a previous company and refused to let it happen again.

The teams that built evaluation frameworks proactively, without an external forcing function, are the minority.

This is not a criticism of the teams that needed the forcing function. The incentive structure does not reward proactive evaluation investment. The teams that built it anyway made a bet that the future cost of not having it would exceed the present cost of building it.

They were right. They are always right.

But being right about a future cost is a hard argument to win against the visible pressure of the current sprint.

The question that reveals everything

Here is the question to ask any AI team before trusting their system.

Not “what is your accuracy?” because they will give you the benchmark number.

Not “how do you test it?” because they will describe their CI pipeline.

Ask this: “Show me your eval set. How many examples does it have. When was it last updated. What happens when a model upgrade causes a regression on it.”

The teams with real evaluation can answer this immediately. They can show you the dataset. They can show you the automated runs. They can show you the regression that a previous upgrade caused and the investigation that followed.

The teams without it will describe a process that exists in intention rather than in practice. They will talk about what they plan to build. They will mention the benchmark they are tracking.

The gap between the answer and the question is the reliability risk you are carrying.

Most teams are carrying more of it than they know.

The model is not the question.

How you know whether the model is working is the question.

And right now, for most teams, the honest answer is: not as well as they should.