The Context Window Just Hit One Million Tokens. Almost Nobody Knows What to Do With It.

A year ago, the most common complaint from teams building on LLMs was that the context window was too small.

Not enough room for the full document. Not enough room for the entire codebase. Not enough room for the conversation history plus the retrieval results plus the system prompt plus the user’s actual question.

The labs heard this. They responded. Context windows expanded from 32k tokens to 128k to 200k to one million and beyond.

The complaint has been addressed.

The teams that were loudest about needing larger context windows are now sitting in front of one million token contexts and discovering something uncomfortable.

Bigger context is not better context. And the difference between those two things is costing companies real money right now.

The attention problem the benchmarks do not show

Every lab publishes benchmarks showing their model performs well on long context tasks.

The benchmarks are real. They are also cherry-picked for the cases where long context works well.

What the benchmarks do not show is what happens to model performance when you fill the context window with real production data rather than carefully constructed test cases.

Real production data is noisy. It contains contradictions. It contains information that was relevant an hour ago and is no longer relevant. It contains information at different levels of importance with no clear signal about which parts the model should weight most heavily.

When you fill a million-token context with this kind of data, the model does not perform like it does on the benchmark. It performs like someone trying to answer a question after reading an entire filing cabinet of loosely related documents.

The answers are plausible. They are not reliably accurate.

The cost nobody calculated

Here is the number most teams discovered only after building with million-token contexts.

Processing one million input tokens with a frontier model costs roughly fifteen dollars per call.

That sounds manageable until you think about what it means at scale.

A product with ten thousand daily active users, each triggering one long-context call per session, is spending one hundred and fifty thousand dollars a day on input tokens alone.

Four and a half million dollars a month. For one feature.

Most teams did not run this calculation before building. They saw the capability, got excited about what it unlocked, and built toward it. The invoice arrived later.

The teams that calculated first built differently. They used large contexts only where large contexts produced meaningfully better outcomes than smaller, cheaper alternatives. The features looked the same to the user. The infrastructure costs were an order of magnitude lower.

What everyone missed about retrieval

When context windows were small, retrieval was necessary.

You could not fit everything in context, so you retrieved the most relevant pieces and included those. Retrieval-augmented generation was the solution to a context constraint.

The assumption was that retrieval would become unnecessary as context windows grew. Fit everything in. No retrieval needed. Simple.

This assumption turned out to be wrong in two ways.

First, the cost problem above. You cannot afford to fit everything in for most production use cases.

Second, and more interesting: retrieval is not just a workaround for small context windows. It is a quality mechanism.

When you retrieve the most relevant chunks for a query and include those, the model works with high-signal information. When you stuff an entire knowledge base into context and ask the same question, the model has to find the signal in the noise. The retrieval step was doing real work that the large context does not automatically replicate.

The teams that abandoned retrieval infrastructure when large contexts arrived are rebuilding it now. The teams that kept it are ahead.

The lost in the middle problem gets worse at scale

There is a well-documented phenomenon in large language model research called the lost in the middle problem.

Models pay more attention to the beginning and end of a context than to the middle. Information buried in the centre of a long context is processed less reliably than information near the edges.

This was a curiosity when context windows were 32k tokens.

It is a serious operational problem when context windows are one million tokens.

The middle of a one-million-token context is five hundred thousand tokens away from either edge. Information placed there is in a kind of dead zone where the model’s attention is at its weakest. For production applications where correctness matters, this is not an acceptable failure mode.

The teams that understand this are structuring their contexts deliberately. The most important information goes at the beginning or the end. The filler, the context that is included for completeness rather than necessity, goes in the middle where the model will weight it less.

Most teams are not doing this. They are treating the context window as a bucket and filling it from top to bottom in whatever order the data arrives.

The outputs reflect this.

The model that reads everything forgets everything

Here is the most counterintuitive finding from teams that have been running million-token contexts in production for several months.

More context does not always mean better recall.

There is a specific failure mode that appears at very long contexts where the model seems to lose track of information it demonstrably has access to. You include a fact in the first ten thousand tokens. You ask about that fact ninety thousand tokens later. The model gets it wrong.

Not because the information was not there. Because the context was so long that the model’s ability to synthesise and recall specific details from across the full window became unreliable.

The researchers call this context saturation. It is not a bug that will be patched in the next release. It is a fundamental property of how attention mechanisms process very long sequences.

The practical implication is that adding more to the context can make specific recall worse, not better. The model is attending to so many things that it attends to each thing less carefully.

This is the opposite of what every team building with million- token contexts assumed they were getting.

The teams winning with large context

The teams using large context windows effectively in 2026 are not the ones that filled them.

They are the ones that built systems that decide what should and should not go into context based on the specific query being answered.

The context is not a static bucket. It is a dynamically constructed input assembled from several sources based on what is actually needed for this specific request. The size of the context varies by query. Some queries need ten thousand tokens. Some need two hundred thousand. The system sizes the context to the task rather than maximising it by default.

This is not a new idea. It is the same idea that RAG was built on, applied more sophisticatedly to an environment where larger contexts are available when they are genuinely needed.

The labs that built the million-token context capability gave teams a larger bucket.

The teams winning with it are the ones that do not always fill the bucket.

What the next six months will reveal

The context window arms race is not slowing down.

Gemini is at two million tokens. Other models are pushing toward ten million. The announcements will keep coming.

Each announcement will be framed as a breakthrough. More context equals more capability equals better products.

The teams that have spent the last year learning the limits of large context windows know that this framing is incomplete.

More context is a capability. It is not a strategy.

The teams that have built the strategy, the systems for deciding what goes in context, in what order, in what quantity, for what queries, will continue to outperform the teams that treat context as a resource to be maximised.

The gap between those two groups is not closing as context windows expand.

It is widening.

Because the teams with the strategy built it in an environment of scarcity, where every token mattered and every inclusion had to be justified. That discipline does not go away when the resource becomes more abundant. It becomes an advantage.

The next capability announcement is coming.

The teams that know what to do with it already are not waiting for it.

They built the answers to that question a year ago, when the context window was still a constraint rather than a commodity.

That is when the real work happened.