Latency Budgets for AI Features: What Users Actually Tolerate
Last updated November 2026
An AI feature’s latency budget sets a waiting-time limit for each interaction, then allocates that time across the request path. Start with what people notice: acknowledge input within 0.1 seconds, deliver something useful within 1 second, and finish or show meaningful progress before 10 seconds. Time to first token and time to completion need separate budgets.
Why an AI feature needs more than one latency target
With a conventional endpoint, you can often focus on when the response arrives. An LLM feature has at least three moments to manage: the interface acknowledges the click, useful text begins to appear, and the answer finishes. Each delay feels different to the person waiting.
Choose the waiting-time allowance before choosing a model. A background summarizer might take 20 seconds without disrupting anyone. An inline autocomplete taking 800 milliseconds can feel unusable. Identical infrastructure can therefore need very different targets depending on the interaction.
The slow requests matter as much as the typical ones. Input size, output length, provider load and retries all change LLM latency. A 1.2-second average can conceal 6-second waits for a meaningful share of users. Set the budget at p95 rather than relying on the mean.
How long will users wait?
Human-perception research is a better starting point than a model leaderboard. In Nielsen Norman Group’s response-time guidance, Jakob Nielsen identifies three limits: about 0.1 second for a response to feel instantaneous, 1.0 second for thought to remain uninterrupted, and 10 seconds for attention to stay on the dialogue. Beyond that, people begin switching tasks.
For interface responsiveness, Google’s web.dev guidance supplies a practical measure: good Interaction to Next Paint (INP) is 200 milliseconds or less; above 500 milliseconds is poor. Measure it at the 75th percentile of page loads. A model may need seconds, but that does not excuse a button that fails to respond promptly.
Waiting can also change later behavior. Google Research’s “Speed Matters” experiment added 400 milliseconds to search results. Searches per user fell 0.44% in the first three weeks and 0.76% in the next three. Even after the delay disappeared, those users made 0.21% fewer searches over the following five weeks. Removing a delay does not immediately undo the habit it created.
| Threshold | Source | What it means for an AI feature |
|---|---|---|
| 0.1 s | Nielsen Norman Group | Feels instantaneous. Target for button state change, input echo, optimistic UI. |
| 200 ms (p75) | web.dev INP “good” | Upper bound for the interface to paint a response to the interaction. |
| 500 ms | web.dev INP “poor” boundary | Past this, the click itself feels broken, regardless of what the model is doing. |
| 1.0 s | Nielsen Norman Group | Flow of thought holds. Target for first visible token or first useful content. |
| 10 s | Nielsen Norman Group | Attention limit. Longer jobs need a percent-done indicator and a way to cancel. |
Set the budget before selecting the model
Write the budget into the design. Use the following sequence when scoping an AI feature.
- Classify the interaction. Is someone waiting in chat, inline suggestions or search; coming back to a draft or document analysis; or never waiting at all, as with enrichment and batch tagging? That distinction determines the ceiling.
- Give every interaction three p95 targets: acknowledgement, first useful output and completion. Streaming may keep the first output quick while the full answer takes much longer.
- Map every hop and assign it both time and an owner: client to API, authentication, retrieval, prompt assembly, model time to first token, generation, post-processing, guardrails and rendering.
- Instrument that real path before tuning it. Collect p50 and p95 using realistic input sizes. Retrieval or a sequential tool call may be slower than the model itself.
- Specify what happens when a hop exceeds its allowance. Choose the timeout, fallback answer, smaller model, cached result or honest error before an incident forces the choice.
- Alert on sustained breaches of the three p95 targets. An acceptable average is not a reason to ignore a broken budget.
Starting targets by feature type
The table gives initial targets based on the perception thresholds above, not universal guarantees. Adjust them using your own measurements; their value is giving the team a concrete starting point.
| Feature type | Acknowledge (p95) | First useful output (p95) | Total completion (p95) |
|---|---|---|---|
| Inline autocomplete or suggestion | Under 100 ms | Under 500 ms | Under 1 s, or drop the suggestion |
| Chat or assistant reply | Under 100 ms | Under 1 s (streamed) | Under 10 s |
| Search answer over your own data (RAG) | Under 100 ms | Under 1 s (results or first sentence) | Under 10 s |
| Agent task with tool calls | Under 100 ms | Under 1 s (plan or first step shown) | Over 10 s allowed, with step progress and cancel |
| Document generation or analysis | Under 100 ms | Under 1 s (job accepted, progress visible) | Asynchronous, notify on completion |
| Background enrichment | Not user-facing | Not user-facing | Set by throughput and cost, not perception |
The acknowledgement target stays the same across user-facing features. Whatever happens behind the interface, the user still needs an immediate response. Meeting that part of the budget is front-end work, and usually the least expensive part to fix.
Where an LLM request spends its time
OpenAI’s latency optimization guide gives seven principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelize, make users wait less, and do not default to an LLM. Its guidance is to generate only the output you need, trim unnecessary input, and measure the effect on your own workload.
Budget completion time accordingly. Limit output length, request structured output and stop generating fields the interface never displays. Prompt trimming can help cost and time to first token, especially when much of the input is new. Our LLM cost control architecture patterns article examines that cost side.
Sequential calls are another common source of delay. Intent classification, retrieval, generation and moderation performed one after another mean four round trips, each with network overhead and its own tail. Parallelize independent work, combine steps where one call can do the job, and use deterministic code for simple tasks. Give retrieval a budget of its own: slow vector queries and excessive context are recurring issues in our discussion of RAG failure modes in production.
Make progress visible without pretending the model is faster
Perceived speed depends on the interface as well as infrastructure. OpenAI recommends streaming so users can begin reading before generation finishes, while noting that time to first visible text still depends on the request, including any reasoning or tool work before the answer. The user starts waiting for the first useful tokens rather than the completed response.
- Stream output longer than a sentence, with the answer first so the opening tokens earn their place.
- Show the actual stage of multi-step work: “Searching 3 sources,” then “Drafting,” instead of one indefinite spinner.
- For jobs beyond 10 seconds, follow Nielsen’s guidance: show percentage or step progress and provide a clear way to interrupt.
- Acknowledge immediately with optimistic UI: disable the button, echo the input and open the response area.
- Remove avoidable work from the critical path by precomputing embeddings, caching frequent answers and enriching data asynchronously.
These choices cannot repair a slow tail. If a chat reply takes 12 seconds at p95, streaming exposes more of the wait rather than eliminating it. Fix the request path before polishing how the wait looks.
Keep the budget intact after launch
Budgets tend to slip through small changes: a longer prompt, a new guardrail in series, or a slow week at the provider. Put all three p95 targets beside error rate on the production dashboard. Alert on sustained breaches and revisit the targets whenever prompts, models or retrieval change. Agents need particular care because every tool call adds variance; our AI agents reliability checklist goes deeper on timeouts and fallbacks.
For a first AI feature, define latency in week one alongside evaluations and cost limits. Our guide to shipping your first production AI feature in 30 days shows where that work fits. For a fixed scope, the $24,500 AI Integration Sprint is run by engineers averaging 10+ years of experience, with the final milestone waived if delivery misses day 30.
Frequently asked questions
What time to first token should an AI feature target?
For chat and search answers, aim for useful output within about 1 second at p95. Nielsen Norman Group identifies that threshold with preserving the user’s flow of thought. Inline suggestions need an even shorter wait.
Does streaming make generation faster?
No. It leaves total generation time unchanged but lets users see the first tokens before the answer finishes. OpenAI’s latency guide recommends displaying text as it arrives, while noting that time to first visible text still depends on the request.
Should the dashboard show averages or p95?
Use p95, adding p99 for high-traffic features. Averages conceal the slow requests caused by input size, output length and provider load. For interface responsiveness, web.dev uses the 75th percentile as its measurement floor.
What is the quickest way to reduce LLM response time?
Start by measuring where users wait, then generate only the output you need and parallelize independent calls. OpenAI recommends comparing time to first token and end-to-end latency on representative requests.
Book a Code Review
Serial calls that could run in parallel, unlimited output, retrieval without a budget and missing timeout behavior are all visible in code. Have a senior engineer review the request path before those weaknesses reach users. Book a Code Review.