Why your LLM feature is slow, and how to reduce LLM latency
Where the seconds go in an LLM feature, how to measure them, and the practical fixes that make AI features feel fast, from streaming to fewer model calls.
Your AI feature works, but users wait. They stare at a spinner, click again, or leave. If you want to reduce LLM latency, the fix is rarely “find a faster model”. Most of the delay usually comes from how the feature is put together around the model.
Here’s how to find where the time goes, and what to change.
Measure before you optimise
“It feels slow” isn’t something you can fix. Log these for every request:
- Time to first token: how long before anything appears on screen.
- Total time: from the user’s click to the finished answer.
- Time per step: retrieval, each model call, tool calls, post-processing.
- Input and output tokens for each model call.
Then look at the slow tail, not just the average. The p95, meaning the request slower than 95% of the others, is what your unhappiest users feel. A trace of a single slow request usually shows the culprit straight away.
Where the time usually goes
- Too many sequential model calls. Classify, then retrieve, then draft, then check, all one after another, each adding its own wait.
- Long outputs. Generating text takes time in proportion to its length. An answer twice as long is roughly twice as slow to finish.
- Bloated prompts. Pages of instructions and twenty retrieved chunks all take time to process, on every request.
- Slow retrieval. Unindexed searches, or calls to several data sources made one after another.
- Waiting for the whole answer before showing anything at all.
Fixes, cheapest first
- Stream the response. Showing words as they’re generated doesn’t make the total time shorter, but it changes how fast the feature feels more than almost anything else.
- Ask for shorter output. Set a clear length in the prompt, ask for structured fields instead of prose where you can, and cap the maximum output.
- Trim the prompt. Remove instructions that no longer earn their place, and retrieve fewer, better chunks.
- Run independent steps in parallel. If two lookups or two model calls don’t depend on each other, don’t make them wait for each other.
- Merge or remove steps. One well-designed call often does the work of two. Simple checks can be ordinary code instead of another model call.
- Use a smaller model where it’s good enough. Routing, classification and extraction rarely need the largest model. Let your evals decide, not your instincts.
- Cache what repeats. Identical questions, common lookups and the stable parts of your prompt are all candidates for caching.
- Move work out of the request. Summaries, embeddings and enrichment can often be done ahead of time or in the background.
Design around the wait that’s left
Some features will always take a few seconds. Make that wait feel intentional:
- Show progress that means something, like “Searching your documents…” and then “Drafting a reply…”, instead of a bare spinner.
- Give a useful partial result early, such as the sources found, while the full answer is still being written.
- For long jobs, let the user carry on and notify them when the result is ready.
Don’t trade quality for speed blindly
Every change above can affect answer quality: a smaller model, a shorter prompt or fewer retrieved chunks. Run your eval set before and after each change, and track speed alongside quality and cost. A faster feature that gives worse answers isn’t an improvement. If you don’t have evals yet, start with how to test an LLM feature. Many of the same levers also lower the bill, as we cover in what an AI feature costs to run.
The short version
- Measure time to first token, total time and time per step, and watch the p95.
- Stream responses, shorten the output and trim the prompt first.
- Run independent steps in parallel, and merge or remove the rest.
- Use smaller models and caching where your evals show quality holds.
- Design the remaining wait so users can see progress.
If your AI feature is slow, expensive or flaky in production, an AI Rescue starts by measuring exactly where the time goes and ends with before-and-after numbers.