KV Cache Explained: The Memory Behind Your AI Bill

You probably picture a rack of GPUs melting through math the moment you send a prompt. For most of the answer, the most expensive chip in the building is barely doing any math at all. It is dragging a scratchpad back and forth from memory, one word at a time, and that scratchpad is the thing deciding how fast the model feels and what each word costs you.

The scratchpad is the KV cache. This article opens it up from the inside: why a model needs a memory at all, where the K and the V come from, why your first word is slow for a different reason than the rest, and how the exact same mechanism turns up on your invoice as prompt caching.

What is the KV cache?

The KV cache is the memory a language model keeps while it generates an answer. Each time it reads a word, it works out what that word means in context and stores the result as two vectors: a Key and a Value. It holds that note for the rest of the conversation, so it never has to work the same word out twice. That growing pile of notes is the cache.

Two things surprise people. The cache is built fresh for every conversation and thrown away when the chat ends, so it is not part of the model. And it holds no text. If you looked inside, you would not find sentences you could read back, just long columns of numbers: what the model made of each word, never the word itself.

Why does an LLM need a memory at all?

Because a model writes one word at a time, and to choose each word it has to look back at everything written so far. The technical name is autoregressive generation, and it traces straight back to the 2017 Transformer paper. The naive way to do it is brutal: write word one and read the whole prompt, write word two and read the whole prompt again, and so on. Every new word re-reads the entire conversation from the top.

Picture reading a thick book where someone stops you after every sentence to quiz you on the whole story. You could jot notes as you go, or you could start over from page one each time. By page three hundred you are re-reading three hundred pages just to add one, and the model slows to a crawl right as the conversation gets good.

Naive recompute

Re-read every earlier word before writing each new one.

word 1

word 2

word 3

word 4

word 5

Work grows with the square of the length. By word 300 it is re-reading 300 words to add one.

KV cache

Work out each word once, write the note, never redo it.

word 1

word 2

word 3

word 4

word 5

Each word costs the same to add, whether you are on word 6 or word 6,000.

So the model does the obvious thing instead. The first time through your prompt, it works out each word’s meaning and writes it down, then never works it out again. Word five’s note reads the same whether the model is on word six or word six thousand. Write once, reuse forever.

Where do the “K” and “V” come from?

Each note has two parts, and that is where the letters come from. The Key is a label for how this word gets found later. The Value is the content the word hands over once it is found. Every word the model has read so far contributes one Key and one Value, sitting in the cache and waiting to be looked up.

There is a third piece that does not get cached. To write a new word, the model forms a Query, a question that means “what am I looking for right now?” It checks that Query against every Key, pulls in the matching Values, and then throws the question away. A Query matters for one word only. The Keys and Values get checked by every word that follows, so they are kept. The model caches the notes and bins the questions: Keys and Values, the KV cache.

Kept in the cache · written once, reused forever

K Key A label for how this word gets found later.

V Value The content the word hands over once it is found.

Thrown away · matters for one word only

Q

Query "What am I looking for right now?" Checked against every Key, then binned.

Prefill and decode: the two speeds you feel

The model works in two phases, and you feel both of them differently. First it fills the notebook. It reads your entire prompt and takes all its notes at once, ripping through the whole thing in parallel, with the chip flat-out busy. This phase is prefill, and the pause it puts between you and your first word is the time to first token, or TTFT.

Then it writes from the notebook, and the mood changes. The answer comes one word at a time, and before each word the model skims every note it holds. This phase is decode, and it is where most of the wall-clock time goes. The split is the basis of modern serving research like DistServe, which pulls the two phases onto separate hardware precisely because they stress the chip in opposite ways.

Prefill Compute-bound

The pause before your first word

• Reads your entire prompt in parallel
• Chip pinned near 100%, flat-out busy
• Sets time to first token (TTFT)

Decode Memory-bound

How fast the answer streams out

• Writes one word at a time, in sequence
• Skims the whole cache for every word
• Chip mostly idle, just moving data

Now back to that idle chip from the start. During decode the most expensive hardware in the building is mostly sitting around. The slow part is not the math, it is hauling that fat stack of notes out of memory, over and over, to write one more word. This is why a long prompt costs you twice: more input means a longer wait for the first word, and a bigger cache to skim for every word after it.

Why the KV cache grows bigger than the model

You never stop taking notes. Every new word adds another, nothing gets evicted until the conversation ends, and the cache only grows. It also grows faster than most people expect. Take a 70-billion-parameter model, around 140 GB of fixed weights. At a long context, the cache for a single conversation already runs past 40 GB, more than half of a top data-center GPU. Stretch that chat toward a million words and the cache sails past 300 GB, roughly twice the size of the model itself.

140 GB Model weights

Fixed for a 70B model. Loaded once and shared across every conversation on the chip. This part never grows.

40 GB+ Cache at 128K

One conversation’s notebook at a long context already eats more than half of a top 80 GB data-center GPU.

300 GB+ Cache toward 1M

Stretch one chat toward a million words and its cache sails past 300 GB, roughly twice the model’s own weights.

Now follow the money. A provider does not hand each customer a GPU. They load the model once and run many conversations on the same chip, because the weights are shared, and that sharing is where the margin lives. But caches are not shared. Every conversation needs its own, so the chip’s memory gets carved up: some for the model, the rest sliced into caches. Pack that space with a few giant caches and you fit fewer people on the chip. Fewer people per chip means each one costs more to serve. That is the cache reaching out of the machine and onto your invoice.

It sets one more limit. Past a certain length, the cache simply will not fit in the memory the chip has. How much a model can actually hold is a separate story about context windows; this one is about what holding it costs.

How does prompt caching cut your AI bill?

Most of what you send an AI barely changes. A giant system prompt, your standing instructions, the documents you paste in front of every question, all of it repeated call after call. Normally the model refills the very same cache from scratch each time, and you pay for that work again. Prompt caching is the deal that fixes it: send a prompt whose opening matches one the provider saw recently, and they keep that filled cache on a shelf instead of binning it. The next call pulls it down ready-made and skips the work.

Cache hit

The prefix matches a recent call, character for character.

• Pull the filled notebook off the shelf
• Skip the note-taking entirely
• Pay only for the new question

up to −90%

Cache miss

One character near the top changed: a date, a name, a space.

• Every note after the change is worthless
• The notebook starts over from page one
• You are billed to refill it

full price

The rule Stable things first (system prompt, fixed documents). Anything that moves, like timestamps or user input, goes last.

Every major provider now offers some version of this, and the savings are real, but the rules differ in ways worth knowing before you architect a prompt.

There is a trap, and people fall in hard. The shelf only works on an exact match, from the very first character. Change one thing near the top, a timestamp, a name, today’s date, and every note after it turns worthless, so the model starts over at full price. One team realised they had been wedging the date and time at the top of every prompt, busting their own cache on every request; moving it to the bottom moved a big chunk of their traffic into cache. Someone else missed it and ran up a $38,000 surprise on AWS Bedrock. The rule that prevents both: stable things first, anything that moves goes last.

How do you shrink the KV cache?

Every limit so far traces to one thing: the cache is too big. The field has spent years inventing ways to shrink it, and four techniques cover most of what runs in production today. The first three happen inside the model and the serving stack; the fourth is one you can flip yourself.

GQA 8× smaller

Grouped-Query Attention lets heads share notes. A 70B model drops from 64 sets to 8, almost for free.

MLA 5–13×

DeepSeek stores a compressed latent and expands it on demand. Its own headline 93% bundles other tricks.

vLLM ~0 waste

PagedAttention hands out memory pages only as they fill, like an OS. The default open-source serving stack.

Quantization 12k → 45k

Fewer digits per number. One dev fit nearly 4× the context on a 24 GB card. Squeeze too hard and quality drops.

Grouped-Query Attention, from a 2023 Google paper, is the simplest: a model looks at each word from dozens of angles and does not need a full note for every one, so the angles share notes. A 70B model that once kept 64 sets can keep eight, shrinking the cache roughly eightfold almost for free. DeepSeek pushed further with Multi-head Latent Attention, storing a compressed shorthand and expanding it only when needed. Their headline 93% reduction bundles a couple of other tricks; on its own the honest range is closer to five to thirteen times, and it is a big reason DeepSeek runs so cheap.

The other two are about organisation and precision. vLLM’s PagedAttention hands out memory in small pages only as they fill, the way an operating system does, cutting the waste that early servers left on reserved-but-empty pages to nearly zero. It is the default way open-source models get served now. Quantization is the lever you control: store each note with fewer bits, and the cache shrinks. One developer flipped an 8-bit KV cache on a 24 GB card and went from about 12K to 45K tokens of context. Squeeze too hard, though, and quality degrades on the long, tool-heavy tasks you wanted the room for. A gentle squeeze is the sweet spot, and it shifts from model to model.

Where KV caching is heading next

The cache is still the frontier, and the newest work is less about shrinking it than deciding who has to carry it. If a hundred people open a chat with the same system prompt, why fill a hundred identical caches? Newer serving engines keep one shared copy and let everyone branch off it, and others park your cold cache in cheaper memory when you step away, then slot it back when you return.

The cache also creates one strange consequence. Your cache lives on one specific chip, so a long conversation is bound to that exact machine. If the system routes your next message to a different server, the normal way the web spreads load, that server has none of your notes and has to read your whole history from scratch. The cache quietly turns a throwaway chat into something with a fixed address, and a lot of old load-balancing wisdom stops working.

The bottom line

The model gets all the attention, but the cache is the thing running the show. It is the model’s memory, the notes it keeps so it never has to re-read your conversation from the top. That is all the KV cache is. But that one idea sits underneath almost everything you notice: the pause before the first word, the way a long chat slows down and runs up the bill, the second identical call that comes back cheap, and the ceiling on how much can fit at once.

None of that is a mystery anymore. It is just a notebook, filling up one note at a time, and now you know who is really doing the heavy lifting. For the full walkthrough with every phase animated, watch the companion video below.