Skip to content
All posts

RAG not working? Chunking, retrieval and reranking mistakes to fix first

When a RAG feature gives vague or wrong answers, the model is rarely the problem. The chunking, retrieval and reranking mistakes to check first, in order.

Shubham Somani 4 min read

Your RAG feature looked great in the demo. Now users say it gives vague answers, misses things that are clearly in the docs, or confidently quotes the wrong policy. The instinct is to try a bigger model or rewrite the prompt. Usually that’s the wrong place to look.

When RAG is not working, the cause is almost always that the right text never reached the model. Here’s where to look, in the order we’d check.

First, find out which half is broken

A RAG system does two jobs: it finds passages, then the model answers from them. Before changing anything, take 20 to 30 real questions where you know the right answer, and for each one check:

  • Did retrieval return the passage that contains the answer?
  • If it did, did the model use it correctly?

If the right passage is missing, no prompt will fix the answer. If it’s there and the answer is still wrong, look at the prompt and the model. This one split saves weeks. We cover building a question set like this in test your LLM feature before users do.

Chunking mistakes

  • Chunks that cut answers in half. Splitting every few hundred characters separates a heading from its paragraph, or a question from the table that answers it. Split on the document’s own structure where you can: headings, sections, list items.
  • Chunks with no context. A chunk that says “this applies to annual plans only” is useless without knowing which document and section it came from. Prefix each chunk with its document title and section heading.
  • One size for every document. A FAQ, a contract and a product manual need different splitting. It’s fine to have more than one strategy.
  • Tables and PDFs flattened into noise. Check what your extraction actually produced. Garbled tables are a common silent failure.

Retrieval mistakes

  • Relying only on semantic search. Embeddings are good at meaning and weak at exact terms like product codes, error messages and names. Combining keyword search with vector search often fixes “it’s right there in the docs, why can’t it find it?”
  • Searching with the user’s raw message. “What about the other one?” retrieves nothing useful. Rewrite the query using the conversation so far before searching.
  • Ignoring metadata. If a question is about one customer, product or region, filter on that before ranking, rather than hoping similarity alone gets it right.
  • Stale or duplicate sources. Old versions of a policy compete with the current one. Remove outdated documents, or store a date and prefer the newest.

Reranking and context mistakes

  • Taking the top few results as they come. First-pass search is built for speed, not precision. A reranking step that re-scores a larger candidate set against the question often lifts the right passage into the top few.
  • Stuffing in too much. More chunks isn’t more accuracy. Long, noisy context can bury the passage that matters and makes every call slower and more expensive.
  • Losing track of sources. Keep the document and section with every chunk so the model can cite them and you can check those citations in code. See reducing hallucinations in production.

When it isn’t retrieval

Sometimes the passage is right there and the answer is still wrong. Then check:

  • The prompt tells the model to answer only from the provided sources, and to say when they don’t contain the answer
  • The question is actually in scope for the feature
  • The sources don’t contradict each other

And if the problem is tone or format rather than facts, retrieval may not be the right tool at all. RAG or fine-tuning covers that choice.

The fix-it checklist

  • Build a set of real questions with known answers
  • Measure retrieval separately from the final answer
  • Split on document structure and add titles to chunks
  • Combine keyword and vector search, and rewrite follow-up questions
  • Filter on metadata and remove stale documents
  • Rerank, then send fewer, better chunks
  • Re-run the question set after every change

If your RAG feature is live and giving answers you can’t trust, our AI Rescue offer is built for exactly this. Or book a 30-minute call and walk us through it.