RAG vs fine-tuning: which one does your use case actually need?

Teams tend to arrive with the question already framed as a technology choice. It is more useful as a diagnosis, and the diagnostic question is short: is the model missing knowledge, or missing behaviour?

What retrieval actually does

Retrieval-augmented generation puts relevant source material into the model's context at request time. The model does not learn anything; it is simply given the facts it needs, in the moment, and asked to reason over them.

This is the right default for most enterprise use cases, for reasons that have little to do with model quality:

• Freshness. Update a document and the next answer reflects it. No retraining cycle.
• Attribution. You can show the user which passage the answer came from, which is often a hard requirement rather than a nicety.
• Access control. Retrieval can filter by the requesting user's permissions. Weights cannot — anything baked into a fine-tune is available to everyone who can call the model.
• Removability. Deleting a document removes its influence. Removing a fact from fine-tuned weights is not a solved problem, which matters when someone exercises a deletion right.

Retrieval is a search problem wearing a new hat

The most common cause of a disappointing retrieval system is not the model. It is that the right passage was never retrieved, so the model was asked to answer from material that did not contain the answer. Before blaming generation, measure retrieval separately.

1. Take fifty real questions. For each, identify by hand which passage contains the answer.
2. Measure how often that passage appears in the top-k results. This is your ceiling — generation cannot exceed it.
3. If recall is poor, fix retrieval: chunking strategy, hybrid keyword-plus-vector search, metadata filters, and a reranking pass over a wider candidate set.
4. Only then tune the generation prompt.

Hybrid search matters more than most teams expect. Pure vector similarity is weak on exact identifiers — part numbers, policy codes, names — which is exactly what enterprise questions are full of. Keyword and vector retrieval combined, then reranked, is the workhorse configuration.

What fine-tuning actually does

Fine-tuning adjusts weights on examples so the model behaves differently by default. It is good at shaping behaviour and poor at installing facts — facts learned this way are unreliable, unattributable, and impossible to update cleanly.

It genuinely earns its place in three situations:

• A rigid output format or domain convention that prompting keeps drifting away from across long outputs.
• A task framing so specific to your domain that explaining it consumes an unreasonable amount of context on every call.
• Cost or latency pressure, where a fine-tuned small model matches a large model on your narrow task at a fraction of the price.

That third case is underrated. If you are running a single well-defined task at high volume, fine-tuning a small model on outputs from a large one is often the highest-return optimisation available.

The third option, which most teams should try first

Before either technique: a better prompt with real examples in it. Few-shot prompting with eight to twelve carefully chosen examples — including the awkward edge cases — closes a surprising share of the gap that teams reach for fine-tuning to fix.

Assembling the examples for a fine-tune is most of the work. Try those same examples in the prompt first — you will often find you are done.

It is also reversible in seconds, requires no training infrastructure, and survives a model upgrade. Fine-tunes have to be redone when you change base models, which in a fast-moving field is a recurring cost.

They are not exclusive

The mature configuration is usually both: retrieval supplies the facts, and a light fine-tune or a well-engineered prompt enforces the behaviour. Diagnose the two problems separately and solve each with the tool suited to it.

Whichever you choose, you cannot tell whether it worked without measurement — see LLM evaluation in practice for how to build the harness that makes these decisions empirical rather than aesthetic.

• RAG
• fine-tuning
• engineering
• architecture