Explainer
Diffusion LLMs Explained: 10x Faster Than GPT?
Mercury Coder writes 1,100 tokens a second, roughly 10x faster than GPT or Claude, by predicting every word at once. How diffusion LLMs work, and where they break.
Watch the explainer
Mercury Coder generates over a thousand tokens per second on a standard H100 GPU. That is roughly ten times faster than GPT or Claude on the same hardware, and not because of a custom chip. It got there by throwing out the one technique every other large language model on the planet depends on.
It does not predict the next word. It predicts all of them, at once. This is a diffusion LLM, and whether you end up using one or not, the ideas behind it are already shaping how the next wave of AI inference gets built. Here is how it works, where it wins, where it falls flat, and why it is worth your attention.
Mercury Coder Mini, independently clocked by Artificial Analysis on a single H100, no custom chip involved
what Inception’s technical report claims against speed-optimized autoregressive models on the same hardware
Mercury’s list price, served behind an OpenAI-compatible API you can point existing code at today
What is a diffusion LLM?
A diffusion LLM is a language model that writes by filling in an entire sequence at once, rather than one word after another. It begins with a row of blanks, predicts every position in parallel, then cleans up its own draft over a few dozen passes. That single change is where the speed comes from.
Every model you have used, GPT, Claude, Gemini, Llama, does the opposite. It writes left to right, one token at a time, and the moment a token is on the page it is locked. A diffusion model treats the whole output as something to sculpt, asking “what should this entire sequence look like?” instead of “what comes next?”
One token at a time, left to right. Once a word is written it is permanent: there is no backspace.
Every position is predicted in parallel, then refined together. The model can revise any word until the last step.
The autoregressive bottleneck, one word at a time
Autoregressive generation, the one-token-at-a-time method, creates three concrete engineering problems, and diffusion was built to attack all of them. The first is speed. You cannot skip ahead, because every token has to wait for the one before it, so throughput is capped no matter how much hardware you throw at it.
The second is editing. Fill-in-the-middle tasks, where you want the model to complete a gap with known text on both sides, are awkward when the architecture only ever looks backward. The third is mistakes. Commit a bad token at position forty-seven and there is no backspace; every token after it has to work around the error. This is the heart of the “diffusion vs autoregressive” question, and it is why the comparison keeps coming up among developers.
How does a diffusion LLM actually work?
It works by masking, not by adding noise. The field first tried to copy image diffusion literally: embed each word as a list of numbers, add a little mathematical noise, then map back to words. That was a dead end. You cannot add 0.3 units of noise to the word “function” and get anything meaningful, and the rounding errors wrecked quality.
What worked was blunter. Instead of noising a token, you replace it with a single [MASK] token, a blank. The training process takes a clean sentence and hides more and more of it. The generation process runs that in reverse: start from a fully masked row, predict every blank at once, keep only the highest-confidence guesses, and leave the rest masked. Repeat for thirty to a hundred passes and the sequence resolves.
Each pass uses bidirectional attention, meaning every position can see every other position, not just the ones to its left. That is what makes fill-in-the-middle native: the model reads the prefix and the suffix at the same time, so editing a gap is just generation with more of the blanks already filled.
Wait, isn’t this just BERT?
For developers, this is the part that should click: a diffusion LLM is basically BERT that learned to write. BERT, the model that powered a generation of search and classification, was trained to predict masked words at a fixed fifteen percent mask rate. A diffusion LLM trains on the exact same masking task, but at every rate from zero to a hundred percent.
That small generalization is the whole trick. BERT only ever saw sentences that were mostly intact, so nobody knew how to make it generate from scratch. Train it to handle any level of masking, including a fully blank input, and you can sample text by starting from all blanks and un-masking step by step. The rigorous version of this argument lives in the MDLM paper, which showed masked diffusion is a weighted mix of masked-language-model losses. Put plainly: BERT was a diffusion language model all along. The field just did not have the recipe to generate from it.
The quality-speed knob autoregressive can’t match
The number of passes is a dial you turn at runtime, and that is something autoregressive models simply do not have. More steps means the model refines its draft longer and the text gets sharper. Fewer steps means it stops early and hands you a rougher answer faster. Same model, same weights, one setting.
Fewer passes. Faster output, lower quality. The same model, dialed down.
More passes. Slower output, sharper text. Trade compute for quality on the fly.
Autoregressive models have no equivalent: every token costs one fixed forward pass, no dial to turn.
An autoregressive model spends one fixed forward pass per token and that is that; there is no way to trade a little quality for a lot of speed on a given request. The diffusion knob is why these models can chase throughput so aggressively, and it is a new degree of freedom in how you serve a model.
Who’s building diffusion LLMs right now?
Three names matter, and they sit at very different points on the open-to-closed spectrum. The commercial leader is Inception Labs, a startup from Stanford professor Stefano Ermon with Andrej Karpathy and Andrew Ng as angel investors. Their Mercury line is the first diffusion LLM you can hit over an API, independently clocked at about 1,100 tokens a second, priced at 25 cents per million input tokens, and ranked fastest on Copilot Arena while tying for second on quality.
Google showed Gemini Diffusion at I/O 2025 with a claimed 1,479 tokens a second, but as of mid-2026 it is still behind a waitlist with no public API, no open weights, and no independent verification. The leaked system prompt reportedly tells the model “you are not an autoregressive language model,” which the community found equal parts funny and philosophically odd.
The first diffusion LLM you can hit over an API. ~1,100 tok/s, OpenAI-compatible, top of Copilot Arena for speed.
Demoed at I/O 2025 with a ~1,479 tok/s claim. Still no public API, no open weights, no independent benchmark.
The reference open model, trained from scratch on 2.3T tokens. Matches Llama 3 8B on many tests; you can run it yourself.
On the open side, LLaDA 8B is the reference model: MIT licensed, trained from scratch on 2.3 trillion tokens. Dream 7B took a shortcut, converting Qwen 2.5’s pretrained weights into a diffusion model with extra training. Both match Llama 3 8B on many benchmarks. Neither cleanly beats Qwen 2.5 7B Instruct on general quality, and that gap is the honest caveat. Apple shipped DiffuCoder 7B for code, ByteDance and Tsinghua published Seed Diffusion, and InclusionAI’s LLaDA 2.x pushed the approach to a 100B-parameter mixture-of-experts. New models keep landing.
Where diffusion LLMs win, and where they break
Diffusion has a handful of clear strengths and several weaknesses, and it shines at a narrow set of jobs today. Speed on long outputs is impressive and independently verified. Code generation is the strongest use case, with Mercury topping Copilot Arena for speed. Fill-in-the-middle is native, and early testers say refactoring is where it shines: hand it a big file, ask it to change the theme, and it edits many spots in parallel.
The weaknesses are just as concrete. Streaming is worse, because an autoregressive model gives you the first token almost instantly while a diffusion model can take a few seconds to show anything, which stings in a chat box. Complex reasoning is weaker, hallucination rates run higher, and the tooling gap is large: there is no mature llama.cpp or vLLM-equivalent path, and the whole agent and fine-tuning stack was built for autoregressive generation.
| Dimension | Autoregressive (GPT, Claude) | Diffusion (Mercury, LLaDA) |
|---|---|---|
| Generation | One token at a time, left to right | Whole sequence in parallel, refined |
| Long-output speed | Capped by sequential decoding | ~1,100 tok/s, up to 10x faster |
| Time to first token | Near-instant, smooth stream | Slower to show the first word |
| Fill-in-the-middle | Bolted on, awkward | Native, sees prefix and suffix |
| Fixing its own mistakes | No backspace once written | Can revise any token until the end |
| Hard reasoning | Stronger today | Weaker, hallucinates more |
| Tooling and ecosystem | A decade of mature stack | Rebuilding from near scratch |
There is an irony the community loves: text generation is drifting from autoregressive toward diffusion at the exact moment image generation is drifting the other way, from diffusion toward autoregressive. The grass really is always greener on the other architecture.
Should you use a diffusion LLM today?
For a narrow set of jobs, yes, and it is a thirty-minute experiment. If you are building code completion, autocomplete, or voice agents where raw output speed is the bottleneck, try Mercury now; it is OpenAI-compatible, so your existing code mostly just works. If you need long context, mature tool use, or frontier-level reasoning, stay autoregressive for now and check back in a year.
The most likely future is not a clean replacement at all. It is a hybrid called block diffusion, where the model moves block to block left to right, like an autoregressive model so it keeps the cache, but fills each block in parallel, like a diffusion model so it keeps the speed.
So is this the moment autoregressive gets replaced? Probably not, at least not soon. But this is no lab curiosity either. Mercury has paying customers, LLaDA is open and MIT licensed, and the ideas behind both are quietly landing inside products you already use. That is what to watch, whether or not you ever call one directly.
Watch the explainer
Frequently asked questions
What is a diffusion LLM?
A diffusion LLM is a language model that writes text by filling in a whole sequence at once instead of one word at a time. It starts from a row of blanks, predicts every position in parallel, then refines over a few dozen passes. That parallel approach is what lets models like Mercury Coder hit over 1,000 tokens a second.
How are diffusion LLMs different from GPT?
GPT and Claude are autoregressive: they predict the next token, left to right, and once a token is written it is permanent. A diffusion LLM predicts every token together and can revise any of them until the last step. That removes the one-at-a-time speed cap and makes fill-in-the-middle editing native rather than bolted on.
Are diffusion LLMs actually faster than GPT?
On long outputs, yes, and it is independently verified. Artificial Analysis clocked Mercury Coder Mini at about 1,100 tokens a second on an H100, and Inception claims up to 10x over speed-optimized autoregressive models. The catch is time-to-first-token: a diffusion model can be slower to show its first word, which hurts in chat.
Is a diffusion LLM just BERT?
Almost. BERT was trained to fill masked words at a fixed 15% mask rate. A diffusion LLM trains on the same masking task but at every rate from 0 to 100%, then learns to sample text by un-masking step by step. BERT was a diffusion language model. The field just did not know how to generate from it yet.
Which diffusion LLMs can I use today?
Mercury by Inception Labs is the main one with a public, OpenAI-compatible API. On the open side, LLaDA 8B is MIT-licensed and runnable yourself, with Dream 7B, Apple DiffuCoder, and LLaDA 2.x as newer entrants. Google Gemini Diffusion remains waitlist-only with no public API or open weights as of mid-2026.
Will diffusion LLMs replace autoregressive models?
Not in the next two to three years. Diffusion wins on speed and code editing but trails on complex reasoning, hallucinates more, and lacks mature tooling. The likely future is hybrid block diffusion: autoregressive between blocks of text, diffusion within each block, which keeps the speed without giving up the cache.
Sources
- Inception Labs · Mercury technical report (arXiv 2506.17298) · retrieved 2026-06-27
- Inception Labs · model list and pricing · retrieved 2026-06-27
- Artificial Analysis · Mercury 2 (throughput and time-to-first-token) · retrieved 2026-06-27
- Google DeepMind · Gemini Diffusion · retrieved 2026-06-27
- LLaDA · Large Language Diffusion Models (arXiv 2502.09992) · retrieved 2026-06-27
- Dream 7B · HKU NLP (Qwen-initialized diffusion model) · retrieved 2026-06-27
- MDLM · Simple and Effective Masked Diffusion Language Models (arXiv 2406.07524) · retrieved 2026-06-27
- Block Diffusion · BD3-LMs, ICLR 2025 (arXiv 2503.09573) · retrieved 2026-06-27
- TechCrunch · Inception raises $50M for diffusion models (founders and investors) · retrieved 2026-06-27
The newsletter
Liked this? Get the Take-Outs.
One email every Tuesday: a spicy take on a trending dev topic, video out-takes, and the tool of the week. Free.