Skip to content
All posts

How to reduce LLM hallucinations in production: a playbook for startups

Hallucinations are an engineering problem, not a mystery. A practical playbook for reducing confident wrong answers in a production LLM feature.

Shubham Somani 4 min read

Every team shipping an LLM feature meets the same moment: a user screenshots a confident, well-written, completely wrong answer. If you want to reduce LLM hallucinations in production, it helps to stop treating them as a mystery of the model. They’re an engineering problem with causes you can find and controls you can add.

You won’t get to zero. You can get to a feature that is wrong rarely, says so when it isn’t sure, and never does damage when it is wrong.

Why models make things up

A language model generates the most plausible next words. It isn’t checking facts unless you make it. Hallucinations usually show up when:

  • The answer isn’t in what the model was given, so it fills the gap with something plausible.
  • The question is outside the feature’s scope, and nothing tells it to decline.
  • Retrieval brought back the wrong documents, and the model answers faithfully from bad context.
  • The prompt rewards an answer over admitting that it doesn’t know.

Most of these are fixable without changing the model.

Ground answers in your own data

  • Give the model the source. For anything factual about your product, policies or customers, retrieve the relevant documents and tell the model to answer only from them. See RAG or fine-tuning for why retrieval is the right tool for facts.
  • Ask for citations. Have the model name the document or passage each claim came from, then check in code that those sources exist and were actually retrieved.
  • Fix retrieval first. If the right passage isn’t in the context, no prompt will save the answer. Measure whether retrieval finds the right document, separately from whether the final answer is right.

Make “I don’t know” an acceptable answer

  • Say it explicitly in the prompt: if the answer isn’t in the provided sources, say so and suggest the next step.
  • Define the scope. List what the feature covers and tell it to decline everything else politely.
  • Include refusals in your evals. If “I don’t know” is never tested, it will quietly stop happening when someone edits the prompt.

Constrain the output

The more freedom the output has, the more room there is to invent things.

  • Use structured outputs like categories, fields and IDs that your code can validate, instead of free text where possible.
  • Check facts in code. If the answer mentions an order number, a price or a date, look it up and compare before showing it.
  • Prefer choosing to generating. Picking from a list of real options is far safer than writing an answer from scratch.

Catch what slips through

  • Add a verification step for high-stakes answers: a second check, by a rule or another model call, that each claim is supported by the sources.
  • Keep a person in the loop where a wrong answer costs money, trust or safety. The model drafts and a person approves.
  • Make it easy to report a bad answer, and add every reported case to your eval set so it can’t quietly return. More on this in how to test an LLM feature.

Measure it like any other bug rate

“It hallucinates sometimes” isn’t a metric. Track:

  • How often answers are supported by the retrieved sources.
  • How often the feature correctly declines out-of-scope questions.
  • How many bad answers users report each week.

Run these on every change to prompts, models or retrieval. A change that makes answers sound better while making them less grounded is a regression, even if the demo looks nicer.

The playbook in short

  • Ground factual answers in retrieved sources, and require citations.
  • Measure retrieval separately from the final answer.
  • Make “I don’t know” an allowed, tested answer, with a clear scope.
  • Use structured outputs and check key facts in code.
  • Verify high-stakes answers or keep a person in the loop.
  • Turn every reported bad answer into a new test.

If your AI feature is giving users confident wrong answers, an AI Rescue starts with exactly this: finding where the bad answers come from, adding the controls, and showing before-and-after numbers.