RAG vs Fine-Tuning vs Prompt Engineering, Explained

Cursor fine-tuned its own AI model to make tab completion feel instant. Sourcegraph, the company behind the Cody coding assistant, published a blog post explaining why it refuses to fine-tune at all. Both were right. That contrast is the whole argument in one line: fine-tuning is not better or worse than RAG or prompting. It solves a completely different problem, and if you pick the wrong one, fine-tuning can make your model dumber, less safe, and more likely to make things up.

So “RAG vs fine-tuning vs prompt engineering” is the wrong fight. They are three different knobs, and the real skill is diagnosing which one your failure needs. This guide walks through what each does, why the advice flipped twice in two years, the three traps that sink most fine-tuning projects, and the rule real teams use to choose.

RAG vs fine-tuning vs prompt engineering: the short answer

Match the knob to the failure. If the model gives wrong or stale information, that is a knowledge problem, so use RAG. If it gives the right answer in the wrong format or tone, or your prompt has grown into pages of rules that still drift, that is a behavior problem, so fine-tune. If you have not exhausted prompting yet, do that first, because it is free and fast.

These three options do not trade off against each other the way “vs” implies. Prompting sets the ceiling of what is possible with a model today. RAG decides what the model knows. Fine-tuning decides how it behaves. Most production systems end up turning all three, because each one fixes a kind of failure the others cannot.

What are the three knobs?

Every LLM app comes down to three levers, and they differ by what actually moves. Prompting rewrites the words going into the model, with no training and no setup. RAG, short for retrieval-augmented generation, leaves the weights alone and instead pulls relevant documents into the prompt at query time. Fine-tuning is the only one that changes the model itself, retraining its weights on your examples until the behavior is baked in.

Prompting

Changes the request

You rewrite the words going into the model. Nothing trains, nothing deploys.

Setup

Seconds

Knowledge

In the prompt

Weights

Untouched

RAG

Changes the context

Your docs sit in a database. At query time the relevant chunks get pasted in before the model answers.

Setup

Hours to days

Knowledge

In the database

Weights

Untouched

Fine-tuning

Changes the model

You train on your own examples until the behavior is baked into the weights themselves.

Setup

A training run

Knowledge

In the weights

Weights

Shifted

The color split holds for the rest of this guide: blue for prompting, yellow for retrieval, red for fine-tuning. RAG is the retrieval half of the picture, and it leans on a vector database to find the right chunks by meaning rather than keywords. With prompting and RAG, the model is a fixed object you feed. With fine-tuning, you are changing the object.

What does fine-tuning actually do?

Under the hood, a model is billions of numbers, and those weights decide which token comes next. When you fine-tune, you show the model examples of the input and the output you wanted, measure how wrong its guess was, and nudge the weights until the answers match. After training, the behavior you were prompting for stops living in every request. It lives in the weights.

That is exactly why Indeed fine-tuned its personalized job-matching model. A prompted GPT model worked, but every call carried a long few-shot prompt, and at their volume that got expensive fast. According to OpenAI’s writeup, Indeed cut prompt tokens by around 80% and ran a smaller fine-tuned model at roughly 60% fewer tokens overall, which let it scale from under a million to about 20 million messages to job seekers a month. The instructions moved out of every request and into the weights, where they cost nothing per call.

Why 2024 said “don’t fine-tune”

For most of 2024 the safe advice was to avoid fine-tuning, and the reason was that teams kept using it for the wrong job. They wanted to bake the company wiki into the weights, to teach the model their internal docs. A 2024 paper from Gekhman at Google found something brutal about that approach: when you fine-tune on facts the base model does not already know, you do not just fail to learn them cleanly, you make the model hallucinate more.

For fresh facts, RAG is the right move. The documents sit in a database, you update the database, and behavior updates instantly, with citations for free and no retraining. This is the exact reasoning Sourcegraph published for why Cody leans on retrieval instead of fine-tuning: a codebase changes every commit, so a model fine-tuned on Monday’s snapshot is stale by Thursday. For anything that moves, retrieval wins.

Why fine-tuning is back in 2026

Then the math flipped, because fine-tuning got cheap. LoRA, short for low-rank adaptation, freezes the giant base model and trains only a tiny patch on top, a few megabytes of new weights. QLoRA squeezed it further by loading the base in four-bit precision, so a fine-tune that used to need a rack of GPUs now fits on one. Meanwhile frontier API prices held roughly flat, so the return on a tuned small model climbed on its own.

Adapter size A few MB

LoRA freezes the giant base model and trains a tiny patch on top, so a fine-tune ships as a few megabytes of new weights, not a fresh copy of the model.

Hardware 1 GPU

QLoRA loads the base in four-bit precision, so a 7B fine-tune that used to need a data center now runs on a single rented GPU in hours, for tens of dollars.

Cost per call 1/40th

On a high-volume narrow task, a tuned small model can match a frontier model while costing a tiny fraction per call. That is the small-model renaissance.

When you serve high volume on a narrow task, a cheap tuned 8B model can beat a general frontier model on that one job while costing a small fraction per call. That is the small-model renaissance, and it is loudest in the open-weight community, where people fine-tune 3B to 8B models and watch them match much larger systems on the work they actually care about. As Hamel Husain puts it, the value is in teaching the model a specific form, not in stuffing it with facts.

RAG vs fine-tuning: which should you use?

Pick by what is failing. RAG is for knowledge that changes or needs to be cited; fine-tuning is for behavior you cannot reliably prompt, like a strict output schema or a consistent tone. The table makes the split concrete, and the honest answer for most teams is that prompting comes first and the other two get added only where prompting runs out.

Notice that prompting and RAG both leave the weights frozen, which makes them cheap to change and hard to break. Fine-tuning is the one with real downside risk, which is why it is the last knob you should reach for, not the first. If you are building agents on top of any of this, the same discipline shows up as harness engineering: the model is one swappable part, not the whole system.

How do real teams split the work?

Look at four production systems and you see four different calls, each driven by a different bottleneck. None of them treated the three knobs as rivals; each picked the tool that matched its actual constraint, and two of them combined more than one.

~260ms Cursor · latency

A keystroke completion has to feel instant, and a frontier model called over the network cannot reliably get there. Cursor trains its own Tab model and reports median latency around 260ms, down from 475ms.

8B beats 340B NVIDIA · distillation

For code-review severity scoring, NVIDIA used a frontier model as a teacher to generate training data, then fine-tuned a small Llama 3 8B student that outperforms models many times its size on that one task.

97% Harvey · all three

The legal AI stacked everything: a base trained on case law, fine-tuned retrieval, and RAG for fresh citations. In blind tests, lawyers preferred its custom model 97% of the time over GPT-4.

Weekly DoorDash · RAG + guardrails

Driver-support policies change every week, so DoorDash keeps them in a RAG layer with guardrail and judge models on top, reporting a roughly 90% drop in hallucinations. Knowledge that moves stays in retrieval.

Cursor fine-tunes its own Tab model because latency is the constraint: a keystroke completion has to feel instant, and a frontier model called over the network cannot reliably get there. NVIDIA took the distillation route, using a frontier model as a teacher to generate training data and fine-tuning a small Llama student that beats far larger models on code-review scoring. Harvey, the legal AI, stacked all three and earned a 97% lawyer preference over GPT-4. DoorDash keeps its weekly-changing support policies in a RAG layer with guardrail and judge models, because policy is knowledge that moves.

The three traps that kill fine-tuning projects

Fine-tuning breaks things in three specific ways, and knowing them is most of what separates a project that ships from one that quietly regresses. All three are documented, not hypothetical, and all three come from the same fact: gradient updates touch more than the behavior you were aiming at.

Forgetting Lost capability

The more you train the model to be great at your task, the more it loses everywhere else. The specialization comes straight out of its general skill, and the usual fixes scale badly.

Hallucination Gekhman 2024

Fine-tuning on facts the base model does not already know increases made-up answers. The model learns to sound confident faster than it learns the content, so the new knowledge leaks out as overconfidence.

< $0.20 Safety regression

A 2023 Princeton study jailbroke GPT-3.5 Turbo with about ten adversarial examples for under twenty cents through the fine-tuning API. Even training on innocent data measurably degraded its refusal behavior.

The first is catastrophic forgetting: the more you train the model to be great at your task, the more capability it loses everywhere else. The second is the Gekhman hallucination problem again, where fine-tuning on unfamiliar facts makes the model more confidently wrong. The third is safety regression, and it is the sharpest: a 2023 Princeton study stripped GPT-3.5 Turbo’s guardrails with about ten adversarial examples for under twenty cents through the fine-tuning API, and even benign training data measurably weakened its refusals. On top of all three, if you cannot score the output, you cannot tell whether fine-tuning helped, so build the eval set before you train.

When should you actually fine-tune?

Diagnose the failure first, then reach for the matching knob. That single habit prevents most wasted training runs. If the failure is wrong information, it is RAG. If it is the wrong shape, wrong tone, or a prompt that has bloated past the point of control, it is fine-tuning. And if you have fewer than fifty good examples and no evals, you are not ready, so prompt harder.

1

Start by prompting. With a strong model it solves most cases for free. Only when prompting stalls do you diagnose which way the failure points.

The failure is wrong information Retrieve for facts RAG

• Stale or missing knowledge
• Needs citations
• Docs change often
• Data is too big to prompt

The failure is the wrong shape Fine-tune for form Fine-tuning

• Right answer, wrong format
• Wrong tone or persona
• Prompt has grown to four pages
• Latency or cost at scale

A few concrete cases sharpen it. If you serve tens of millions of calls a month on a stable task and prompt length is eating your budget, fine-tune a smaller model and put the prompt into the weights, the way Indeed did. If latency is the constraint and retrieval is too slow, fine-tune, the way Cursor did. And if your domain has patterns the base model just does not know, like legal reasoning or a proprietary schema, fine-tune for the form and add RAG for the facts, the way Harvey did.

The verdict

Every production LLM app turns three knobs. Prompting sets the ceiling of what is possible today, RAG decides what the model knows, and fine-tuning decides how it behaves. The question stopped being “which one do I pick” and became “which layer is my bottleneck,” because most real systems end up using all three. The Cursor team and the Cody team reached opposite conclusions about fine-tuning, but they agreed on the method: diagnose the failure, then reach for the matching knob.

If there is one line to keep, it is the community mnemonic: fine-tune for form, retrieve for facts, and prompt for everything you can get away with first.