How to test an LLM feature before your users do
Most AI features ship on vibes: someone tries five prompts and says it looks good. A small eval set turns that into a number, and it takes less time than you think.
Here’s how most AI features get tested. Someone on the team types five or six questions, reads the answers, and says “looks good”. Then it ships, a user asks the seventh question, and the answer is confidently wrong.
You wouldn’t ship a payments change that way. An LLM feature deserves the same discipline, and the tool for it is boring: an eval set.
What an eval set is
An eval set is a list of inputs your feature will see, each paired with what a good output looks like. You run the feature against every example, score the results, and get a number. Change the prompt, the model or the retrieval, and you run it again. If the number goes down, you don’t ship.
That’s it. It’s a regression test suite for behaviour you can’t write a simple assertion for.
Start with 30 real examples, not 3,000
The most common mistake is waiting until you can build a big, perfect dataset. Don’t. Thirty well-chosen examples will catch most regressions:
- 10 happy paths. The questions you built the feature for.
- 10 hard cases. Ambiguous questions, long inputs, questions that need two documents to answer.
- 10 things it must not do. Questions outside its scope, attempts to get other customers’ data, requests it should refuse or hand to a human.
Take them from real usage where you can: support tickets, sales calls, the questions your team answers by hand today. Invented examples test what you imagined. Real ones test what users do.
Decide what “good” means before you score
For each example, write down what a correct answer must contain or must avoid. Then pick the cheapest scoring method that works:
- Exact checks where possible. Did it return valid JSON? Did it pick the right category? Did it cite a document that exists?
- Simple rules next. Does the answer mention the refund window? Does it avoid giving medical advice?
- A model as judge for the rest, with a written rubric, spot-checked by a person. Useful, but never the only check.
If two people on your team disagree about whether an answer is right, that’s not an eval problem. It’s a product decision you haven’t made yet, and it’s far cheaper to make it now than after launch.
Run it on every change
An eval set you run once is a report. One you run on every change is a safety net. Wire it into CI:
- Every prompt change, model swap or retrieval tweak runs the evals.
- The pull request shows the score next to the previous one.
- A drop below the agreed threshold blocks the merge, just like a failing test.
Track latency and cost per request in the same run. A change that improves accuracy by 2% and doubles the cost is a decision, not a free win.
Keep feeding it
Every time a user reports a bad answer, add it to the eval set with the correct answer. Within a few weeks your evals reflect your real users better than any dataset you could have bought, and the same mistake can’t quietly come back.
The boring checklist
Before an LLM feature ships:
- 30+ real examples, including the things it must not do.
- A written definition of a good answer for each.
- Scoring that runs automatically, cheapest method first.
- Evals in CI, with a threshold that blocks the merge.
- Latency and cost tracked alongside quality.
- A habit of turning every bad answer into a new test.
None of this is exciting. It’s what lets the exciting part ship without a crossed-fingers emoji in the release notes.
If you have an AI feature that works in the demo and wobbles in production, an AI Rescue starts with exactly this: an eval set, a baseline, and before-and-after numbers.