What is multi-token prediction (MTP)?

Multi-token prediction is a technique where a language model predicts several future tokens at once instead of one at a time. Extra lightweight prediction heads, trained alongside the main model, draft the next few tokens, and the full model verifies them in a single pass. It speeds up inference up to 3x with no change to the output.

Is multi-token prediction the same as speculative decoding?

They are closely related but not identical. Speculative decoding is the general inference trick of drafting tokens cheaply and verifying them in one pass. MTP is the version where the draft heads are trained into the model itself, rather than running a separate small draft model. MTP is built-in speculative decoding.

Does MTP reduce the quality of the output?

No. A rejection sampling step guarantees the model produces exactly the same text it would have without MTP. Drafts that the full model would not have chosen are rejected and replaced from its own distribution. It is mathematically identical output, just generated faster. This is not an approximation or a shortcut.

How much faster does multi-token prediction make inference?

It depends on the workload. Structured, predictable text like Python or JSON sees around 3x because the draft heads guess it cleanly, with roughly 90% acceptance. General chat lands near 2 to 2.5x, and creative writing or translation closer to 1.8x. The more predictable the next tokens, the bigger the gain.

When does multi-token prediction not help?

MTP struggles on sparse mixture-of-experts models, because verifying drafts can force the GPU to load different experts per token and wipe out the gains. It also fails on busy servers at high concurrency, where the GPU is already compute-bound, and on tiny 1B to 3B models that are already fast enough.

Multi-Token Prediction (MTP), Explained Simply

A 27-billion-parameter model on a single RTX 3090, running at 50 tokens per second, with no quality loss and no new hardware. That number would have gotten you laughed out of a Reddit thread six months ago. Right now thousands of developers are hitting those speeds on the same GPU they bought two years ago, thanks to something called multi-token prediction.

Most people still do not really understand how it works, or why it makes inference so much faster without changing a single word of the output. This guide walks through what MTP actually does, the memory bottleneck it exploits, the real speedups by workload, and the three setups where it quietly stops paying off.

What is multi-token prediction?

Multi-token prediction is a technique that lets a language model generate several tokens per step instead of one at a time. During training, the model gets extra output heads, small prediction modules bolted on top of the main network, each taught to guess one token further into the future. At inference, those heads draft the next few tokens and the full model checks them all in a single pass.

The payoff is speed with no catch. Meta’s 2024 paper, Better & Faster Large Language Models via Multi-token Prediction, reported that models trained to predict four tokens at once ran up to 3 times faster at inference, and the same training objective made a 13B model solve 12% more problems on HumanEval. The output you get is exactly what the model would have produced one token at a time. It just arrives sooner.

Why your GPU is bored, not slow

The reason MTP helps comes down to a bottleneck most people get backwards. When a model generates text, it produces one token at a time, and for every single token the GPU loads tens of gigabytes of model weights out of memory and into its compute cores. The math at the other end finishes almost instantly. The slow part is the memory transfer, not the arithmetic.

So the GPU’s compute mostly sits idle, waiting on data. This is the memory bandwidth bottleneck, and it is why a card with 165 teraflops of raw compute gives you only about 25 tokens per second on a 70B model. The hardware is not slow, it is starved.

95%

Memory bus pegged moving 40GB of weights per token

Compute idle, the math finishes in under 1ms

165 TFLOPS gives you 25 t/s on a 70B model. The hardware is not slow, it is starved for data.

Fig. 1 The throughput gap MTP exists to close. The memory bus runs near flat-out moving weights while the compute cores sit nearly idle. All that raw power converts to a token rate that feels slow.

Every standard decode step wastes that idle compute. If you could verify several tokens during one trip through the weights, you would be putting hardware you already paid for back to work. That is the whole opening MTP walks through, and Google frames it the same way in its Gemma 4 writeup: standard inference is memory-bandwidth bound, which is exactly the gap a drafter fills.

How does multi-token prediction work?

The trick is to draft cheaply, then verify in bulk. The lightweight heads race ahead and guess a few tokens, say A, B, C. The full model then takes all three candidates and verifies them in a single forward pass: one weight load, one trip through tens of gigabytes, but potentially three or four correct tokens out instead of one.

It starts at training time. Alongside the normal next-token head, the model grows a handful of extra heads that each predict a token further ahead. Forcing it to look multiple steps into the future also makes it learn deeper, longer-range patterns, which is why the training objective improves quality on its own. Then those same heads become a built-in draft system at inference.

Fig. 2 The architecture trick. A shared transformer trunk forks into prediction heads. Head 1 is the normal next-token head; heads 2 to 4 each guess one token further ahead and become the draft system at inference time.

If all the drafts are correct, and on code they are right roughly 90% of the time, you triple your output for the same memory cost. If one is wrong, the model keeps the tokens up to the mistake, emits the correct token from its own distribution, and the heads start over. Worst case, you wasted a little draft compute and still got one correct token, the same as a normal step.

All drafts correct

A ✓ B ✓ C ✓

1 weight load → 3 tokens. Verified together in a single forward pass. On code this happens ~90% of the time, so you triple output for the same memory cost.

Second draft wrong

A ✓ B ✕ C ✕

Keep A, drop the rest. The model emits the correct token from its own distribution and the heads restart. Worst case you still get one token, exactly like a normal step.

Fig. 3 The draft-and-verify loop, both outcomes. Left: all three drafts pass in one forward pass, so one weight load yields three tokens. Right: the second draft is wrong, so the model keeps A, replaces the rest, and restarts.

The part that makes this safe is a rejection sampling step. It guarantees the model emits the exact same text it would have without MTP. Not an approximation, not a shortcut, not a quality trade. The same output, just faster.

Multi-token prediction vs speculative decoding

This is the question developers keep asking, and the two terms get conflated constantly. Speculative decoding is the general idea: draft tokens with something cheap, then verify them with the full model in one pass. MTP is the specific version where the draft system is trained into the model itself, as extra heads, instead of running a separate small “draft model” alongside the big one.

That difference matters in practice. A separate draft model is another set of weights to load, host, and keep aligned with the main model. MTP heads share the main model’s trunk, so they are tiny, always in sync, and ship inside one file. The table lays out where they diverge.

Dimension	Classic speculative decoding	Multi-token prediction (MTP)
Who drafts	A separate small draft model	Extra heads inside the main model
Training	None, you bolt on any small model	Heads are trained with the model
Alignment	Drafter can drift from the target	Always in sync, shares the trunk
Overhead	A second model to load and host	A few lightweight heads, one file
Output	Identical (rejection sampling)	Identical (rejection sampling)
Side benefit	None for the base model	Better training, deeper patterns

So MTP is not a rival to speculative decoding, it is the cleanest way to do it. The model ships with its own drafter baked in. That is also why MTP rode the same wave as approaches like Medusa and EAGLE, which attach trained heads to existing models, and it pairs naturally with the kind of sparse mixture-of-experts designs shipping in frontier open models.

How much faster is MTP, really?

It depends on what you are generating, and the pattern is intuitive once you see it. Code is where MTP shines: Python, JSON, anything structured and predictable, the draft heads nail those patterns and stack up acceptance streaks of four or five tokens in a row. Community benchmarks on Qwen 3.6 27B show around 3x on Python at roughly 90% acceptance.

General chat lands closer to 2 to 2.5x, and creative writing or translation around 1.8x. The more predictable the next few tokens, the longer the streaks and the bigger the speedup. A surprising word breaks the streak, the model resets, and no harm is done.

Python / JSON

~90% draft acceptance

General chat

2–2.5x

mixed predictability

Creative / translation

1.8x

streaks break often

More draft tokens does not mean more speed. Speedup grows roughly logarithmically, so 3 to 5 draft tokens is the sweet spot for most workloads. Past that you burn compute for almost nothing.

Fig. 4 MTP's speedup tracks predictability, not a single fixed number. Structured code drafts cleanly and streaks; prose surprises the heads and resets more often. More draft tokens helps only logarithmically.

There is a hard ceiling, though, and it is not a tooling limit but a mathematical one. A 2025 theoretical paper, Speculative Decoding Speed-of-Light, proved that speedup grows only logarithmically with the number of draft tokens. You cannot draft 20 tokens and expect 20x. Three to five draft tokens hits the best trade-off for most workloads; beyond that you burn compute for diminishing returns.

The MTP ecosystem in 2026

A year ago MTP was a research paper. Today it ships in production models. DeepSeek-V3 trained with a native MTP objective across its 671B parameters, Qwen 3.5 and 3.6 did the same, and in May 2026 Google released dedicated MTP drafter models for Gemma 4, claiming up to 3x speedup with no quality loss. The drafter is now a standard part of how open models ship.

The tooling caught up just as fast. vLLM and TensorRT-LLM support it in production, Ollama added it, and llama.cpp, the engine behind most local setups, gained MTP support in a beta that developers hammered on for days. Apple Silicon turns out to be a star here, because its unified memory bandwidth plays directly into MTP’s strengths.

~20 → 50 t/s

RTX 3090

A two-year-old card runs a 27B model at coding-assistant speed. This is the benchmark that started the buzz.

5 → 12 t/s

MacBook Air M4

A fanless laptop crosses from unusable to functional on a 27B model. Unified memory bandwidth plays right into MTP.

28 → 63 t/s

M5 Max

A different tier of usable. Fast enough that local inference stops feeling obviously worse than a cloud API.

Fig. 5 MTP is a consumer-hardware story. Before and after token rates on Qwen-class 27B models. Numbers are community benchmarks; treat them as the shape of the gain, not a guaranteed spec for your setup.

For the first time, developers running 27 to 31B models on a single consumer GPU sit in the 25 to 60 tokens-per-second range. That is fast enough for a responsive coding assistant and for agentic tool-calling loops with rapid back-and-forth, the workloads where local inference used to feel obviously worse than a cloud API.

Where does multi-token prediction fall short?

MTP is not a universal 3x, and the gotchas matter before you rebuild your stack around it. The biggest one is sparse mixture-of-experts models. When you verify draft tokens on an MoE model, each token might route to different experts, so the GPU has to load different weight slices for each one, which kills the whole point. One thorough benchmark on a Qwen 3.6 MoE variant on a single 3090 found every speculative configuration was slower than baseline, even at 100% acceptance.

High concurrency breaks it too. Above batch sizes of 16 to 32 the GPU is already compute-bound, so there is no idle capacity left to exploit. MTP is a latency trick for a single user, not a throughput trick for a busy server. And on tiny 1B to 3B models that already run at 100+ tokens per second, the drafting overhead becomes a noticeable fraction of each step.

MoE models 0.89×

Each draft token can route to different experts, so the GPU reloads weight slices. One 3090 benchmark went slower than baseline even at 100% acceptance.

High concurrency > 16

Above batch sizes of 16 to 32 the GPU is already compute-bound. MTP is a latency trick for one user, not a throughput trick for a busy server.

Tiny models 1–3B

Models already running at 100+ t/s gain little. The drafting overhead becomes a real fraction of each step, so you add cost to something fast enough.

Fig. 6 The three setups where MTP stops paying off: sparse mixture-of-experts models that reload experts per token, high-concurrency servers that are already compute-bound, and tiny models that were already fast enough.

How do you try MTP today?

If you run a dense model in the 27 to 31B range on a consumer GPU or Apple Silicon, you are the ideal case, and the setup is close to one flag plus one model file. Pick a model that ships MTP heads, like Qwen 3.6 27B or a Gemma 4 drafter pair, run it through vLLM, Ollama, or a recent llama.cpp build, and enable speculative decoding. Keep the MTP heads at Q8 or better.

If you are on a sparse MoE model, a tiny model, or serving high-concurrency traffic, measure before you commit, because MTP may do nothing or even slow you down. For everyone else, the upside is free: the same model, the same output, materially faster.

The GPU’s compute power was always there. MTP just found a way to put it to work, and with every major open model now shipping MTP heads and every major framework adding support, this is not on the way. It is already here.

Watch the explainer