Context Window Explained: Why 1M Tokens Still Forget

When Google shipped Gemini 3 Pro, the model card said one million tokens. Paying subscribers ran their own tests in the web app and measured the usable window at about thirty-two thousand. Not a typo: thirty times smaller than the headline. Every frontier model has the same gap, and most of them admit it in a footnote or a price surcharge.

This article is about that gap. What a context window really is, how attention reads it, why a million-token model still forgets halfway through your document, and what working developers do instead of trusting the number on the box.

What is a context window?

A context window is the fixed-size buffer a model reads while it computes one answer. It is not memory and it is not a notebook. Every token your prompt contains has to fit inside that box: the system message, the chat history, the file you pasted, the tool output your agent just generated. All of it competes for the same space, and when the box is full, the model cannot see anything else.

Two things trip people up. The advertised number is usually input plus output combined, so a long reasoning trace eats into your own context. And the chat interface is faking continuity. There is no state between requests, so the platform pastes the whole conversation back in on every single turn. That feeling that the model remembers you is a transcript on rewind.

FIXED WINDOW · decided by the hardware before your prompt ran

System prompt

Chat history

Pasted file

Tool output

free

Outside the window earlier turns, evicted · invisible to the model

Why doesn’t a bigger context window mean a better answer?

Because the advertised number is the size of the room, and the number you actually get is the patch of floor where the model can still walk in a straight line. Anthropic admits the gap in its system cards. Google admits it in footnotes. OpenAI admits it with a surcharge that doubles the input rate past a threshold. The marketing leads with capacity; competence is a much smaller number.

Advertised 1 million tokens

The number on the model card. The size of the room.

30× gap

Actually usable ~32K tokens

Measured in the web app. The patch of floor the model walks straight on.

So treat the headline the way you would treat a laptop spec. Yes, the 32GB of RAM is in the machine. No, you cannot fill it with garbage and expect the thing to run faster. The rest of this article is about why that floor sits so far below the ceiling.

Self-attention and the shrinking slice

The mechanism that reads the window is self-attention. Every token asks every other token how relevant it is, those scores get normalized into weights that sum to one, and the model mixes information in proportion to those weights. That last detail is where it breaks.

The weights sum to one no matter how many tokens you have, so attention is a fixed pie. The more tokens you add, the thinner the slice each one gets. At a thousand tokens a key sentence might pull thirty percent of the attention mass. At a million, that same sentence is fighting 999,999 neighbours for the same finite pie. The model does what you would do at a party with a million people: nod politely at everyone, listen to nobody.

1,000 tokens ~30%

the key sentence can pull a third of the attention mass

1,000,000 tokens ~0.003%

the same sentence fights 999,999 neighbours for the same finite pie

Position is the other half of the problem. Transformers are order-blind by construction, so modern models patch that with rotary position embeddings (RoPE), which rotate each token’s vector by an angle based on where it sits. That works well inside the length the model was trained on and worse and worse outside it. Most million-token models were trained on far shorter sequences and stretched up to the headline number with tricks that buy capacity, not fidelity.

The race from 2,000 to 10 million tokens

Five years ago GPT-3 shipped with a 2,048-token window, about the length of this section read aloud. In May 2023 Claude jumped to 100K. In February 2024 Gemini 1.5 Pro hit a million. In April 2025 Llama 4 Scout claimed ten million. Today every frontier model ships with at least a million.

1. Jun 2020 2,048 GPT-3
2. May 2023 100K Claude
3. Feb 2024 1M Gemini 1.5 Pro
4. Apr 2025 10M Llama 4 Scout
5. 2026 1M+ every frontier model

No single trick did this. RoPE rescaling stretched position encodings after training. FlashAttention cut the memory bill. Sparse attention skipped most of the pairwise comparisons. Hybrid architectures mixed transformer layers with state-space layers that scale linearly instead of quadratically. The product pressure was simpler than any of that: customers wanted to paste in their codebase, their contracts, their whole knowledge base, and stop building retrieval pipelines. Vendors heard that and started racing on the headline.

Why does a long context window still forget?

Because the failure is not one thing. Long context breaks in five different ways at the same time, and each one alone costs only a few points. Stacked together, they are the reason a million-token model loses the plot at hour three of an agent session.

U-curve Lost in the middle

Tokens at the start and end get attended to; the middle gets skipped. MIT showed in 2025 it is structural to causal masking, not a bug you can retrain away.

18 / 18 Context rot

Chroma tested 18 frontier models in 2025. Every one degraded as input grew, even on tasks they solved trivially at short length.

99% → 70% Keyword shortcut

On NoLiMa, GPT-4o fell from 99% at 1K tokens to 70% at 32K, on the same task, once the answer stopped sharing keywords with the question.

15% vs 90% Multi-needle

At 128K tokens, Llama 4 Scout scored 15% on narrative comprehension; Gemini 2.5 Pro scored 90% on the exact same test. Two facts and a link between them breaks models.

whisper Instruction drift

Deep in an agent loop, the system prompt is still in the window, but its weight against piled-up tool calls keeps dropping. The recent turns shout; the rules whisper.

The first two, lost in the middle and context rot, are big enough to have their own names and their own sections below. The other three are quieter. The keyword shortcut means attention has an easy hook when your question and the answer share words, and falls apart when they do not: the NoLiMa benchmark caught GPT-4o dropping from 99% accuracy at 1K tokens to 70% at 32K on the same task, just by removing that overlap. Multi-needle reasoning breaks the moment you need two facts and a relationship between them rather than one. And instruction drift means that deep into an agent loop, the system prompt is still in the window but keeps losing weight against the volume of tool calls piling up around it.

What is context rot?

Context rot is the steady drop in quality as the input grows, well before you hit the advertised limit. In July 2025 Chroma tested 18 frontier models, Claude, GPT-4.1, Gemini 2.5, Qwen, and every one degraded as input length grew, even on tasks they solved trivially when the input was short. A 200K window can rot badly at 30K.

The weirdest finding from that study: shuffling the context randomly sometimes made performance better. A coherent, logical document gives the model more plausible-looking distractors to confuse itself with than a pile of unrelated paragraphs does. The takeaway is blunt. The window filling up is not the danger; the window filling up with relevant-looking noise is.

What is “lost in the middle”?

Lost in the middle is the U-shaped accuracy curve that long context follows: information at the start and the end of the window gets attended to far more than information in the middle. Liu et al. showed that GPT-3.5 performed worse with the right answer buried in the middle of the prompt than with no documents at all. The model’s version of skipping to the intro and the conclusion and pretending it read the book.

It is not a training bug you can fix with more data. In 2025 researchers showed the bias is structural to how causal masking moves information through a deep stack of layers. Bigger windows do not cure it; they widen the U. The middle just gets longer.

Context window sizes by model

Most 2026 frontier models advertise at least a million tokens, and the model-card number is the one that ends up in marketing. The table below pairs each advertised window with what independent testing shows, because those are rarely the same figure.

The pattern repeats across every vendor. A “claude context window” or “gemini context window” search returns a clean round number, and the round number is the ceiling, not the working space. The single-needle “needle in a haystack” tests that vendors quote are saturated at 99% and measure lexical retrieval at one position. Real tasks need multi-hop reasoning, and that breaks far earlier.

What long context actually costs you

Long context bills you three ways, and only one of them is obvious. The price cliff is the visible one: OpenAI doubles the input rate above 272K tokens, Google charges more above 200K, and Anthropic flat-rates to a million, which makes it the exception rather than the rule. Latency is uglier, since prefill on a million tokens takes many seconds before you see a single output token.

Price cliff 2×

OpenAI doubles the input rate above 272K tokens, for the rest of the session. Google charges more above 200K.

Re-sent history £121

Ten short turns on a 700K-token history in Gemini AI Studio, which re-submits the whole conversation every turn with no caching by default.

Gone before turn 1 35%

Share of a 200K coding-agent window eaten by the system prompt, tool definitions, and CLAUDE.md before you type a word.

The trap most people miss is the re-sent history. One developer fed Gemini AI Studio a series of short prompts on top of a 700K-token conversation, and ten turns later the bill was £121, because AI Studio re-submits the entire conversation every turn with no caching by default. Prompt caching helps, but only when the prefix is byte-identical between requests. Reorder a tools array, shuffle your retrieved documents, change one character in the system prompt, and you cache-miss and start paying from scratch.

How do developers handle long context?

The working playbook is the opposite of “paste everything and let the model sort it out.” It is five habits, repeated on every coding-agent forum, and they all push in the same direction: keep the window small and clean.

01

Stay under 30%

Quality drops visibly past ~30% of the window on agent tasks. The cap matters more than the headline size.

02

Filter, then load

A small, focused RAG prompt usually beats stuffing everything in. Irrelevant tokens do not just cost money, they make the model dumber.

03

Files, not chat

Put decisions in a file the agent can re-read on demand: CLAUDE.md, a decisions log. Chat history is the worst place to store state.

04

Use subagents

Spawn a subagent with its own fresh window and a narrow task, then have it report back a one-paragraph summary. The orchestrator stays clean.

05

Restart, don't recover

The three-strike rule: if the agent cannot fix it in three tries, the context is already polluted with failed attempts. Kill it, start fresh.

The headline rule is to stay under about 30% of the window, because quality drops visibly past that on agent tasks. Filter first and load second, since a small retrieval prompt usually beats a giant dump and irrelevant tokens actively make the model worse. Put decisions in files the agent can re-read on demand rather than trusting the chat history. Use subagents with their own fresh windows for narrow tasks, and have them report back a one-paragraph summary. And restart instead of trying to recover: if the agent cannot fix something in three tries, the context is already polluted with its own failed attempts, so kill it and start fresh.

The number on the box and the number you get

The advertised window is the size of the box. The effective window is the depth at which the model still attends, retrieves, and reasons cleanly, and in 2026 those are not the same number. The gap is wider than the marketing implies, and until vendors lead with the second figure, you are going to have to measure it yourself.

Next time a vendor leads with one million tokens, treat it like a hardware spec. The capacity is real. Filling it with noise and expecting a better answer is not how the machine works. For the full walkthrough, with the benchmarks and the failure modes animated, watch the companion video below.