"Should we fine-tune a model on our data?" is probably the question we hear most. In most cases the answer is no — but not for the reason people expect.
Two different problems
RAG gives the model access to knowledge: your documents, your procedures, your current data. Fine-tuning changes the model's behavior: its output format, its tone, its command of specialized vocabulary, its adherence to internal conventions.
The decision criterion fits in one sentence: if the right answer changes when your documents change, it is a knowledge problem — RAG. If the right answer must be phrased in a way the base model does not naturally produce, it is a behavior problem — fine-tuning.
Why fine-tuning is not a knowledge base
A model fine-tuned on your documents does not "know" them the way a database does. It has absorbed statistical regularities. It can restate an internal policy correctly nine times and invent the tenth — and nothing distinguishes the two answers from the outside.
Worse: your documents evolve. Every significant update means another training run, another evaluation, another deployment. A RAG index updates in seconds.
Where fine-tuning wins
Three cases come up constantly: the output must follow a strict format that prompting cannot reliably produce; the domain vocabulary is so specific the base model mishandles it; or latency and cost demand a small model, distilled for your task, which then beats the large generalist.
That third case is the most profitable and the most ignored. An 8-billion-parameter model specialized on your classification often beats a generalist twenty times its size — at a hundredth of the inference cost.
The default combination
In practice our architectures almost always land in the same place: RAG for knowledge, a base model for reasoning, and light fine-tuning only where format or cost demand it. Start with RAG. Add the rest when measurement justifies it.
