Latency, trust and the two variables everyone conflates
Slower can be more trusted, and there is research on why
Thesis
In 2011, Ryan Buell and Michael Norton ran a study on travel search. They built versions of a search tool that showed their work: the sites the engine was querying, the results being filtered, the process visibly happening. Users preferred those versions over faster ones that simply returned an answer. They rated the slower, more transparent tool as producing better results and were willing to wait longer for it.
That finding should be uncomfortable for anyone who treats latency as a number to minimise, and it points at the reason latency discussions in AI products go nowhere.
Latency is two variables pretending to be one. There is duration, which is how long the operation takes, and it runs into perceptual limits that were measured before most of us were born. And there is legibility, which is how much the user understands about what is happening, and it determines whether they trust the result.
Duration is an engineering problem with known thresholds. Legibility is a design problem with no thresholds at all. They can move in opposite directions, and confusing them produces products that are fast and distrusted or slow and forgiven.
Context
Inference has put a large class of products into a response-time band that user interface design has almost no experience with.
The pre-AI world had two normal cases. Local operations that completed inside a tenth of a second, which felt instant. And genuinely long jobs, measured in minutes, which everyone understood to be asynchronous: you started them, you left, you came back.
Model calls land in between. Two seconds. Eight seconds. Forty seconds for an agent running a few tool calls. That band is uniquely bad. It is far too long to feel like direct manipulation, and far too short to justify leaving the screen. The user sits and watches, which is the worst thing a user can do with their time, and it is now the dominant interaction pattern in an entire product category.
Most teams respond to this by trying to shave seconds. That work matters and it has a ceiling. The larger available gain is in the second variable.
Research
Miller, 1968, "Response time in man-computer conversational transactions". The original source of the thresholds everyone quotes secondhand. Miller identified distinct response-time regimes for different kinds of interaction, and his framing has survived sixty years of hardware change because the limits are human rather than technical.
Card, Robertson and Mackinlay, 1991, and Nielsen's 1993 restatement. The three figures worth memorising. Below roughly 0.1 seconds, the system feels like an extension of your hand and direct manipulation is possible. Up to about 1 second, thought is uninterrupted, though the user notices the delay. Beyond about 10 seconds, attention leaves and the user starts doing something else, at which point you owe them a way back.
Doherty and Thadani, 1982, "The Economic Value of Rapid Response Time". IBM's finding that dropping system response below roughly 400 milliseconds produced disproportionate gains in user productivity, because at that point the interaction stops being a sequence of requests and becomes continuous work. This is the threshold that matters for anything a professional uses all day, and it is far stricter than most teams assume.
Maister, 1985, "The Psychology of Waiting Lines". Eight propositions from service operations, every one of which applies to a loading state. Unoccupied time feels longer than occupied time. Uncertain waits feel longer than known, finite waits. Unexplained waits feel longer than explained ones. Anxiety makes waits feel longer. Solo waits feel longer than group waits.
Read that list and then look at an undifferentiated spinner. It is unoccupied, uncertain, unexplained and solo. It is the worst possible design under every proposition simultaneously, and it is the industry default.
Buell and Norton, 2011, "The Labor Illusion". The travel search finding above, published in Management Science. Operational transparency increased perceived value and willingness to wait. Buell, Kim and Tsay extended this in 2017 to show reciprocal effects when both sides of a service can see each other working.
The boundary condition is important and often dropped when this study gets cited: the effect depends on the labour being believable and the domain being one where effort plausibly improves the result. Showing work on a task users expect to be instant does not build trust. It looks broken.
Harrison and colleagues, 2007 and 2010, on progress bars. Progress bars with the same total duration are perceived as faster or slower depending on their behaviour. Bars that accelerate toward completion feel shorter than linear ones; bars that pause or slow near the end feel much longer. Perceived duration is partly a rendering decision, which is a strange and useful fact.
Argument
Get into a better band rather than shaving seconds inside a bad one.
The single highest-leverage change in most AI products is not reducing total completion time. It is reducing time to first visible output.
Token streaming is the obvious instance and it is worth being precise about why it works. A forty-second completion streamed with a first token at 400 milliseconds is not a forty-second wait that the user tolerates better. It is a 400 millisecond wait followed by forty seconds of reading, and reading is occupied time. Maister's first proposition does the work here. You have not made the operation faster. You have moved the user out of waiting entirely.
The same logic generalises past text. Return the interpretation of the request immediately, before the result exists. Render the shape of the answer, then fill it. Show the retrieved sources while the synthesis runs. Any of these converts dead time into something the user can read, evaluate or correct.
Where duration genuinely cannot be hidden, make the wait known and finite.
Uncertain waits feel longer than known ones. A spinner communicates neither duration nor progress, which means it is maximally uncertain by construction.
An agent doing multi-step work has real, nameable phases: reading the document, querying three sources, drafting, checking. Naming the current phase costs one line of state and converts an opaque wait into visible progress. Adding a count makes it finite: step two of four. Now the user knows how long, roughly, and what for.
This is also where the honest version of the labor illusion lives. You are not manufacturing the appearance of effort. You are reporting effort that is genuinely occurring, which happens to be what users need in order to calibrate their expectations of the output.
Optimistic feedback, with a hard condition attached.
Acknowledge the action instantly, in the interface, before the server confirms it. This is the cheapest way to stay under 100 milliseconds for the part the user's hand is connected to.
The condition, and it is not optional: if confirmation does not arrive, the state must visibly revert and say so. Optimistic feedback without rollback is not a performance technique, it is a lie with a latency benefit. Users discover it eventually, usually at the worst moment, and what they lose is not patience but trust, which is a much more expensive thing to lose.
Waiting creates a debt that the output has to repay.
This is the part of the latency conversation that gets missed entirely, and I think it is the most important.
An instant wrong answer costs the user a second and some annoyance. A forty-second wrong answer costs them forty seconds they spent believing something useful was coming. The reaction to the second is categorically different, and it is not proportional to the extra time. It reads as a betrayal rather than an error, because the user made an investment on the strength of an implied promise.
The practical consequence is that latency raises the accuracy bar. If your product takes thirty seconds to answer, it needs to be right more often than a product that answers instantly, not equally often. And the corollary is sharper: never make a user wait for something they did not ask for. Unrequested latency is pure cost with no possibility of repayment, which is why an automatic AI action that delays the interface is worse than the same action offered as a button.
Latency budgets belong to the interaction, not to the endpoint.
Engineering teams measure per-request latency. Users experience per-task latency, which includes every request in the sequence plus their own thinking time.
A product where each call takes 1.5 seconds and a task takes nine calls is a thirteen second product, and nobody is measuring the thirteen. Fewer round trips beats faster round trips, and the design decision that removes a step is usually worth more than the engineering work that shaves 200 milliseconds off each remaining one.
Cost and latency trade against each other, and the tradeoff is a product decision.
Smaller models are faster and worse. Larger ones are slower and better. This is a real dial and it should be set per task rather than per product, which means someone has to decide which operations need to feel instant and which are allowed to take time.
The general shape that works: anything in the user's direct manipulation path gets the fast model or no model at all. Anything the user explicitly asked for and expects to think about gets the good one. Getting this backwards, which is common, produces a product that is slow at the things that should be instant and mediocre at the things that should be careful.
Examples
A search interface over documents. Return the matching documents in 200 milliseconds from the index, then stream the synthesis. The user is reading source titles while the model works, which is occupied time, and they can abandon early if the sources are wrong. This is strictly better than a five second wait for a synthesised paragraph, even though the total duration is identical.
An agent that books or purchases something. Show the steps as they happen, and stop before the irreversible one. The visible sequence is doing two jobs at once here: it occupies the wait and it gives the user the information they need to authorise the final action. Transparency and safety turn out to be the same design.
Autocomplete or inline suggestion. This must be under 100 milliseconds or it is worse than nothing, because the user's typing has already moved past the point the suggestion applies to. If your model cannot hit that, use a smaller one, cache aggressively, or do not ship the feature. A late suggestion is an interruption.
A creative tool with real-time playback. Here the two variables split hardest. Playback is a hard real-time constraint measured in milliseconds, and no amount of transparency fixes a dropout. Generation is a several-second operation where transparency is the whole game. Those are two different latency regimes inside one product, and the architecture has to keep them apart, because the generation path cannot be allowed to touch the audio path.
A form field validating against a model. Do not. Validate deterministically, and if a model has to be involved, do it on blur rather than on keystroke. Nothing degrades a typing experience faster than a probabilistic check in the input loop.
Counterargument
"This is all compensation for bad engineering. Make it faster and none of it is needed."
The strong version of this is correct on its own terms. Every technique above is a mitigation, and a genuinely fast product needs none of them. Speed is the real answer: better caching, smaller models where adequate, precomputation, parallel calls, streaming at the infrastructure level rather than the interface level. A team that spends its effort on progress-bar psychology instead of on the p95 has chosen the easier work, and users would rather have the faster product than the better-explained slow one.
There is a second objection, on ethics, and it deserves more respect than it usually gets. Deliberately showing work to increase perceived value is a manipulation of perception rather than an improvement in substance. The Buell and Norton result is, read uncharitably, a finding about how to make people value something more without making it better. Designing around it puts you in the same territory as artificial loading animations in financial products, where the delay exists to signal diligence that is not occurring.
Where this is right. Speed first, always. If you can get under a second, do that instead of anything in this article, and the ordering is not close. And the ethics point draws a line I would keep: reporting real work is transparency, and inserting delay or fabricating steps to imply effort is a lie. The difference is whether the interface is describing something that is actually happening. A progress step that names a query you really ran is honest. A three-second "analysing" animation over an instant operation is not.
Where I think it is wrong. Inference latency is not primarily an engineering failure. A large model producing a long output has a floor set by the physics of sequential generation, and no amount of infrastructure work removes it. Telling teams to just be faster, in that band, is telling them to solve a problem that is not theirs.
And the streaming objection cuts the other way in one specific case worth naming: streaming is wrong for output the user will not read linearly. Structured data, code intended for pasting, or anything the user will act on as a whole should appear complete or not at all, because partial structured output invites the user to start acting on something that is going to change.
Practical implications
Measure time to first visible output separately from total completion. Optimise the first one first. Most teams do not have a metric for it.
Delete every undifferentiated spinner. Replace with a named phase and a count. Step two of four, drafting.
Stream anything read linearly. Do not stream anything acted on as a unit.
Acknowledge under 100 milliseconds, always, even if the acknowledgement is only that the request was received.
If you use optimistic UI, build the rollback in the same commit. Not the next sprint.
Budget latency per task, not per request. Count the round trips in a real workflow and reduce the count before reducing each one.
Set the model tier per operation. Fast or absent in the manipulation path, capable in the deliberate path.
Raise the accuracy bar with the duration. A slow feature needs to be right more often than a fast one to be worth keeping.
Never impose latency on an action the user did not request.
Report real work only. If the phase label does not correspond to something happening, remove it.
My perspective
Opinion, from my own work.
The hardest latency problem I have encountered was not a model call. It was the seam in a music production tool between real-time audio, where the budget is a few milliseconds and a miss is audible, and inference, where the budget is seconds. Those two regimes cannot share a path. The audio thread cannot wait for anything, and the generation path cannot pretend to be instant.
What I learned there is that the answer is architectural rather than perceptual. You separate the paths completely and you let the interface show two different kinds of time. Real-time work gets no progress indication at all, because indication implies waiting and there is none. Generative work gets full transparency, because it is a request with a result. Blending them, by putting a subtle spinner into a transport control for instance, degrades both.
I should also say what I am not claiming. There is a latency improvement figure attached to my Veena Studio work in various places on my own site, and I have excluded it from this archive because I cannot evidence it. The same percentage appears attached to a completely different metric elsewhere in my materials, which is a good indication of where it came from. I would rather write two thousand words on latency with no number of my own in them than reuse a figure I cannot stand behind.
The thing I believe most strongly and can prove least: perceived performance is a trust instrument, not a comfort feature. Users do not form a view about whether a product is fast. They form a view about whether it is competent, and speed and legibility are the two largest inputs to that judgement, ahead of output quality in the first thirty seconds of use. Which means the loading state is doing more work than almost anything else on the screen, and it is usually the last thing anyone designs.
Conclusion
Duration and legibility are separate variables. Duration has hard limits established by Miller in 1968 and unchanged since, and inference sits in the worst available band: too slow for direct manipulation, too fast to leave the screen.
Legibility is where the available gains are. Move users out of waiting by showing partial output. Make unavoidable waits known and finite by naming real phases. Acknowledge instantly and roll back visibly. And remember that a wait is an investment the user made on your promise, which means the output has to be worth it, and unrequested latency can never be.
Make it faster where you can. Where you cannot, make it legible, and do not confuse the two jobs.
Sources
- R. B. Miller, "Response time in man-computer conversational transactions", AFIPS Fall Joint Computer Conference, 1968.
- S. K. Card, G. G. Robertson and J. D. Mackinlay, "The information visualizer, an information workspace", CHI 1991.
- J. Nielsen, "Response Times: The 3 Important Limits", Nielsen Norman Group, 1993. https://www.nngroup.com/articles/response-times-3-important-limits/
- W. J. Doherty and A. J. Thadani, "The Economic Value of Rapid Response Time", IBM, 1982.
- D. H. Maister, "The Psychology of Waiting Lines", in The Service Encounter, 1985.
- R. W. Buell and M. I. Norton, "The Labor Illusion: How Operational Transparency Increases Perceived Value", Management Science 57(9), 2011, 1564–1579.
- R. W. Buell, T. Kim and C.-J. Tsay, "Creating Reciprocal Value Through Operational Transparency", Management Science 63(6), 2017.
- C. Harrison, B. Amento, S. Kuznetsov and R. Bell, "Rethinking the Progress Bar", UIST 2007.
- C. Harrison, Z. Yeo and S. E. Hudson, "Faster Progress Bars: Manipulating Perceived Duration with Visual Augmentations", CHI 2010.
Related reading in this archive
- Designing products when AI is part of the interface
- Agents, autonomy and the human in the loop
- Veena Studio: the real-time and inference seam
- Cognitive load is the real SaaS design problem
- Designing trust in AI interfaces: the reliance literature the trust claims above are read from