Skip to content
All posts

RAG or fine-tuning? How to teach an LLM about your business

Most startups that ask for fine-tuning need retrieval instead. Here is how to tell which one your feature needs, and what to build first.

Shubham Somani 4 min read

Sooner or later every team building with LLMs hits the same wall. The model is clever, but it doesn’t know your product, your customers or your policies. Someone suggests fine-tuning a model on your data. Someone else says “just use RAG”. Both sound reasonable, and they solve different problems.

Here’s the short version: retrieval gives the model knowledge, fine-tuning gives it habits. Most startup features need knowledge first.

What each one actually does

Retrieval-augmented generation (RAG) finds the relevant pieces of your data at request time and puts them in the prompt. The model reads them and answers from them. Your help centre, contracts, product docs or past tickets stay in a store you control, and the model only sees what each request needs.

Fine-tuning trains a model further on examples of inputs and the outputs you want. It changes how the model behaves: its format, tone, or the way it handles a narrow task. It is a poor way to store facts, because the model can’t tell you where a fact came from, and updating it means training again.

Reach for retrieval when

  • The answer depends on facts that change. Prices, policies, inventory, documentation. Update the source and the next answer is current, with no training run.
  • You need to show your sources. Retrieval lets you cite the exact document or paragraph, which users and compliance teams both want.
  • Different users can see different data. Permissions are easy to enforce at retrieval time and close to impossible to enforce inside a model’s weights.
  • You’re still learning what users ask. A retrieval pipeline is quick to change. A fine-tuned model is a slower thing to iterate on.

Consider fine-tuning when

  • You need a consistent format or style that careful prompting and a few examples still don’t produce reliably.
  • The task is narrow and repeated at high volume, such as classifying tickets into your categories, and a smaller tuned model could do it faster or cheaper than a large general one.
  • You already have hundreds of good examples of the exact input and output you want, and an eval set to prove the tuned model is better.

If you can’t tick that last point, you aren’t ready to fine-tune. You’d be training on data you haven’t checked, with no way of knowing whether it helped.

Try the cheaper fixes first

Before either, check that the problem isn’t the prompt. A clear instruction, the right context and two or three worked examples fix a surprising number of “the model doesn’t understand us” complaints. Then:

  1. Prompt and examples. Hours of work, easy to change.
  2. Retrieval. Days to a couple of weeks for a solid first version.
  3. Fine-tuning. Only once 1 and 2 are in place and your evals show a gap they can’t close.

They also combine. A common mature setup is retrieval for the facts plus a lightly tuned model for the format, but very few teams need the second half on day one.

Where retrieval goes wrong

RAG is not magic either. When it disappoints, the cause is usually upstream of the model:

  • Messy source data. Outdated docs, duplicates and contradictions get retrieved and repeated faithfully.
  • Bad chunking. Splitting documents in the wrong places separates a question from the paragraph that answers it.
  • No retrieval evals. Teams test the final answer but never check whether the right document was found. Measure both.
  • Too much context. Stuffing twenty chunks into the prompt costs more and often answers worse than five good ones.

The short version

  • Retrieval for knowledge, fine-tuning for behaviour.
  • Fix the prompt before reaching for either.
  • Start with retrieval for anything built on your own documents or data.
  • Fine-tune only with real examples and evals that show it helps.
  • When RAG underperforms, look at the data and the retrieval before blaming the model.

Not sure which one your feature needs? A two-week AI Sprint answers exactly that question, with a working prototype and evals to back the decision.