Should you fine-tune a model or use RAG?

The distinction that decides it

RAG adds knowledge. At request time you retrieve relevant documents and put them in the context, and the model answers from them. Knowledge is external, updatable and citable.

Fine-tuning adjusts behaviour. You train on examples so the model reliably produces a particular shape of output — a format, a tone, a classification, a domain convention.

Almost every question reduces to: is the problem that the model does not know something, or that it does not behave how you want?

When RAG is right

• The information changes. Policies, prices, inventory, docs.
• The information is private and per-customer.
• You need citations. Users need to see where an answer came from, and a fine-tuned model cannot tell you.
• The corpus is large. Retrieval scales with a vector store; training does not scale with corpus size the same way.
• You need to remove something. Deleting a document removes its influence immediately, which matters for correction and for compliance.

That last point is underrated. A fine-tuned model that learned something wrong requires retraining to unlearn it.

When fine-tuning is right

• Format compliance. You need exact structured output every time and prompting gets you to 95%.
• A narrow, high-volume classification. A small fine-tuned model can be far cheaper and faster than a large model with a long prompt.
• A style or voice that is tedious to specify and easy to demonstrate.
• Latency and cost pressure where a smaller model, tuned, matches a bigger one prompted.
• Domain convention — a specialist register that prompting keeps drifting away from.

Try prompting properly first

A significant share of fine-tuning projects are solving a prompt problem. Before training anything: give clear instructions, provide several examples in context, constrain the output format explicitly, and break the task into steps.

This is not a fallback — it is the cheapest experiment available, it takes hours instead of weeks, and the resulting prompt is a better specification of what you wanted than the fine-tune would have been.

The maintenance difference

RAG's ongoing cost is retrieval quality: chunking, embeddings, ranking, and keeping the index fresh. It is real work, and it is the usual cause of a RAG system that answers fluently and wrongly.

Fine-tuning's ongoing cost is that the artefact ages. Base models improve, and a fine-tune on last year's base is a decision you have to periodically remake. Products that fine-tuned heavily on early foundation models found the ground shifting under them as the base models absorbed the capability they had trained for.

Start with RAG plus a well-built prompt. Fine-tune when you have a specific, measured behaviour problem that prompting demonstrably cannot fix — and keep evals running either way, because both approaches fail quietly.