Agents, autonomy and the human in the loop
Four positions, and three of them are usually theatre
Thesis
"Human in the loop" has become the phrase teams reach for when asked how their agent is safe, and it has stopped carrying information. It appears in launch posts, in procurement answers and in regulatory filings, and in almost none of them does it say where the human is, what they see, what they can do about it, or whether anyone has checked that they do.
That vagueness is not harmless, because there are only a handful of places a human can actually sit in an agent loop and they behave completely differently.
You can put the human on the goal, on the plan, on each action, or on the result. Those are four different products with four different failure modes. The default choice in most agent tooling is on each action, which feels like the most cautious option and is frequently the least effective, because it produces a person clicking approve without reading.
And the thing that should determine the choice is not how much you trust the model. It is how reversible the action is and how quickly a mistake becomes visible.
Context
The framing exists because agents changed what software does between two user inputs. A form submits and something specific happens. An agent takes an instruction and then performs an open-ended number of steps, some of which touch the world, before returning.
Everything in the human-factors literature about supervising automation suddenly applies to ordinary product design, and most product teams have not read it. This is a field with fifty years of results on exactly this problem, mostly from aviation, process control and medical devices, and it says several inconvenient things about oversight that the current generation of agent products is rediscovering slowly and at the user's expense.
The inconvenient headline: adding a human to a loop does not reliably add safety. It adds a person, and whether that person catches anything depends on details of what they are shown and how often.
Research
Sheridan and Verplank, 1978, levels of automation. The original ten-level scale, from a computer that offers no assistance through to one that acts entirely on its own and tells the human nothing. Parasuraman, Sheridan and Wickens extended this in 2000 into a model that separates the stages automation can occupy: information acquisition, information analysis, decision selection, and action implementation.
That second distinction is the useful one and it is largely missing from product conversations. An agent can be fully automated at analysis and fully manual at action. Treating autonomy as a single dial is the mistake the model exists to prevent.
Endsley and Kiris, 1995, on the out-of-the-loop performance problem. In their study, participants using higher levels of automation were slower to recover when the automation failed, and their situation awareness was measurably lower. The mechanism is not laziness. Passive monitoring produces a weaker mental model of the system state than active control does, so when the human is finally needed they have to reconstruct the context before they can act.
This is the finding that undermines the simple "human can always intervene" story. The human can intervene. They intervene worse than they would have if they had been doing the task.
Skitka, Mosier and Burdick, 1999, on automation bias. They separate two error types: omission errors, where the human fails to notice something because the automation did not flag it, and commission errors, where the human follows an automated recommendation against other available evidence. Both increased in the presence of an automated aid, including in participants who were explicitly told the aid was fallible.
Applied to an approval dialog, this predicts something specific and unwelcome. A human asked to confirm an agent's proposed action is measurably more likely to approve it than they would be to choose it themselves.
Mackworth, 1948, the clock test, and the vigilance literature that followed. Detection of rare signals degrades within the first half hour of monitoring, and the effect is robust across sixty years of replication. Sustained attention to infrequent events is a task humans are poor at, and no amount of instruction fixes it.
An agent that asks for approval forty times an hour has built a vigilance task and will get vigilance-task performance.
Bansal and colleagues, CHI 2021, "Does the Whole Exceed its Parts?" They tested whether AI explanations help humans form better joint decisions. The result was that explanations increased the rate at which people accepted the model's recommendation, whether or not the recommendation was correct, and did not reliably improve team accuracy.
This is the most important result for anyone designing agent review interfaces, and it is the opposite of what most teams assume. Showing the reasoning is not a control. It is a persuasion surface, and it persuades in both directions.
Breznitz, 1984, and the cry-wolf effect. Repeated false alarms degrade response to true alarms. Well-established across alarm research, and it converts an apparently conservative design choice, flagging anything uncertain, into an actively unsafe one.
Green, 2022, on policies requiring human oversight of government algorithms. A review of oversight mandates in public-sector algorithmic systems, arguing that they frequently fail on their own terms: people cannot perform the evaluation the policy assumes, and the requirement's main effect is to legitimise the system while providing little protection. His term for it is a false sense of security, and the pattern transfers directly to commercial products that list human review as a feature.
Argument
The four positions are genuinely different products.
On the goal. The human states the objective and the constraints, then does not participate until the end. Cheapest, and correct when the whole task is reversible and errors are visible on inspection.
On the plan. The agent produces a plan, the human reads and edits it, then execution runs. This is the most undervalued position. The human reviews once, with full context, at a moment when nothing has happened yet, and the artifact being reviewed is small enough to actually read.
On each action. The agent asks before every consequential step. This is where the literature above lands hardest: vigilance decrement, automation bias, cry-wolf, all at once. It is not useless, but its value collapses as the number of prompts rises, and it is the only position that gets worse the more you use it.
On the result. The agent acts, the human reviews output with a full record and a reversal path. Correct when actions are cheaply reversible and mistakes are detectable, and it is the position where humans perform best, because reviewing a concrete artifact is a much easier task than authorising a hypothetical one.
Reversibility and detection latency pick the position. Model quality does not.
Two questions decide it. Can this be undone, and at what cost. If it goes wrong, how long until anyone knows.
Cheap to undo and fast to detect: act, then show. A draft, a local file edit, a layout change. Gating these is where approval budget gets wasted.
Cheap to undo but slow to detect: act, then show, and add a scheduled or diff-based review, because "reversible" is worthless if nobody notices for three weeks. Quiet data corruption lives here and it is the case teams handle worst.
Expensive to undo and fast to detect: gate it, at the plan or at the action. Sending an email, posting publicly, moving money.
Expensive to undo and slow to detect: this is the category that should get a hard stop and a second party. Deletions, permission changes, anything that touches money without a receipt the user reads.
Note what is absent from all four: how good the model is. Reliability changes how often you get to the bad branch. It does not change what happens when you do, and the design has to be correct on the branch.
You have an approval budget, it is small, and every low-stakes prompt spends it.
Treat meaningful human approvals as a scarce resource, on the order of a handful per session. Spend them on the irreversible.
That implies batching. Group the reversible actions and report them together afterwards. Gate the small number that matter, and make those prompts visibly different from ordinary confirmations, in wording and in weight, because a user who has approved nine trivial things will approve the tenth without reading it.
Make review cheaper instead of more frequent, and accept that the review interface is the product.
This is where agent products are won and it is consistently the least designed surface. The options are known and mostly unused: a diff rather than a description, a dry run that reports what would change, a plan as an editable artifact, a grouped summary organised by consequence rather than by chronology, and a persistent log that can be read after the fact by someone who was not watching.
One prompt on a good diff beats twenty prompts on prose summaries. And given the Bansal result, the summary the agent writes about its own work should be treated as an advocacy document rather than as evidence.
Explanations are not oversight, and confidence scores are not either.
If your safety story is "we show the model's reasoning", the CHI 2021 result says you have built something that increases acceptance rather than accuracy. Reasoning traces are useful for debugging and for teaching. They are not a control, and they should not be counted as one in any internal document.
Keep the human doing enough of the task to stay competent at it.
The Endsley and Kiris finding has an uncomfortable product implication: full automation of a task degrades the human's ability to take it back, which matters most precisely in the systems where taking it back is the whole safety argument.
For consequential workflows this argues for deliberately retaining some human execution, or for periodic manual passes, even where automation performs well. It is a real cost paid for a real capability, and it should be an explicit decision rather than an accident.
Design the appeal path before the automated decision.
If an agent can decline, block, flag or restrict a person, someone has to be able to contest it, and the route has to exist before the feature ships. An automated decision with no appeal path is not an autonomy level. It is an unaccountable one, and retrofitting the route after complaints arrive means the first affected users get nothing.
Examples
A coding agent. Reversible, fast to detect, version-controlled. Correct position is on the plan and on the result: review the plan, let it run, review the diff. Per-action approval on file writes is the industry default and it is mostly ceremony, since the actual review happens at the diff and every prompt before it dilutes attention.
An agent that sends email on your behalf. Irreversible and immediately visible. Gate every send, individually, forever. This is what the approval budget is saved for.
An agent doing bulk data cleanup. Reversible in principle, slow to detect in practice. Dry run first with a full report, then execute, then a diff review. The dangerous version of this feature is the one that succeeds quietly.
Automated fraud or risk decisions. Expensive to undo, and the cost falls on someone who did not choose the automation. Needs a human path for the contested cases and an appeal route designed at the same time as the model, not after.
A research agent that only reads. Fully autonomous is correct. Gating a read is pure friction, and the review that matters is of the conclusions, which is a different activity from approving steps.
Counterargument
"This is transitional scaffolding. Model reliability is rising fast, and products that bet on elaborate human oversight will lose to products that simply work."
The strong version has real force. Every checkpoint is a place a user stops, and abandonment at a confirmation dialog is a measurable product cost paid on every run, while the error it prevents happens rarely. If a competitor removes the dialog and their error rate is acceptable, they win on the experience and the safety argument turns out to have been a tax.
There is a sharper version. Elaborate oversight design is a way for a team to feel responsible without being accountable, and Green's own argument can be turned around and pointed here: a carefully designed four-position framework is exactly the kind of artifact that produces a false sense of security. The honest alternative is to measure the error rate and publish it.
And a third, from the other direction: the aviation and process-control literature comes from environments with trained operators, certification regimes and hours-long shifts. Consumer software has none of that, so importing findings from it is analogy rather than evidence.
Where this is right. The trajectory point is correct and the checkpoint set should be treated as revisable rather than permanent. Any gate that has been approved a thousand times without a single rejection is not a control, it is a habit, and it should be removed and replaced with after-the-fact review. Teams that add checkpoints and never audit them end up with the cry-wolf problem they were trying to avoid.
The accountability point is right too, and it is the better critique. Measured error rates would tell you more than any framework here, and I do not have them.
On the domain-transfer objection: partly conceded. The vigilance and automation-bias findings are laboratory results with trained or at least instructed participants, and the effect sizes should not be assumed to carry over precisely. The direction is what I am relying on.
Where I think it is wrong. Reliability and consequence are independent, and only one of them is improving. A ninety-nine percent reliable agent that can wire money is not safer than a ninety percent one, in the way that matters, because the loss on the bad branch is unchanged and the human has now stopped reading the prompts. Endsley and Kiris plus the vigilance literature predict that improving reliability degrades oversight quality, which means the gate on genuinely irreversible actions is the one thing that should not be relaxed as models improve.
The correct response to rising reliability is therefore asymmetric: delete gates on reversible actions aggressively, and keep the small set that guard irreversible ones indefinitely.
Practical implications
Write down your autonomy level per action type, not per product. One dial for the whole agent is the original error.
Classify every action on two axes: cost to undo, and time to detect. The position follows from the classification.
Count your prompts per session. If it is above a handful, you have built a vigilance task and you are getting vigilance performance.
Batch the reversible, gate the irreversible, and make the irreversible prompts look and read differently from ordinary confirmations.
Invest in the diff, the dry run and the plan artifact before adding another confirmation step. Cheaper review beats more review.
Do not count reasoning traces or confidence scores as controls. They increase acceptance, which is not the same as increasing accuracy.
Add a slow-detection review for anything reversible but quiet. A weekly digest of what the agent changed is worth more than a real-time prompt nobody reads.
Design the appeal path at the same time as the automated decision, and staff it.
Audit your gates quarterly. Any gate with no rejections in a thousand runs should be replaced with after-the-fact review.
Keep some human execution in consequential workflows, on purpose, to preserve the competence the intervention story depends on.
My perspective
Opinion, and it comes with one design failure of mine that fits the argument uncomfortably well.
I use agentic coding tools daily. Claude Code, Cursor, and n8n for automation. My honest observation from that, offered as one operator's experience rather than as evidence: the review that catches anything is the diff, and the per-file approvals before it catch almost nothing. I have approved things I did not read, in sequences where I had already approved eight similar things, and the reason was not carelessness. It was that the ninth prompt carried no information the first eight had not already carried.
That is the automation-bias and cry-wolf pattern arriving in my own workflow, and noticing it is why I now think plan review is the underrated position. Reviewing a plan is a task I do properly. Reviewing the four hundredth confirmation is a task I do badly, and I do it badly in a way that improves as the tools get more reliable, which is the wrong direction.
The failure. TapQR's published positioning names three pillars, and one of them is AI fraud detection. That layer is not built, and the honest documentation of the project says so. But the design as specified had a hole in it that has nothing to do with implementation: I described automated fraud decisioning on merchant transactions without designing any path for a merchant to contest a decision. A system that can flag a small merchant's payment, with no appeal route, is a decision about someone's money made by a component with no accountability surface attached.
I did not omit the appeal path because it was hard. I omitted it because I was designing the capability and not the consequence, which is exactly the mistake this article is about. If that pillar ever gets built, the review queue and the contest route get designed first, and the detection model second.
Conclusion
The phrase means nothing on its own. Ask where the human sits: on the goal, on the plan, on each action, or on the result. Each is a different product and each fails differently.
Choose by asking how expensive the action is to undo and how long it takes anyone to notice a mistake. Reversible and visible, act and report. Irreversible or quiet, gate it, and gate it at the plan where the human can still read what they are authorising.
Spend approvals like the scarce resource they are, make review cheap rather than frequent, and stop treating explanations as oversight. And as the models get better, delete the gates that protect against recoverable mistakes while keeping the few that protect against the ones you cannot take back. Reliability improves the odds. It does not change the loss.
Sources
- T. B. Sheridan and W. L. Verplank, "Human and Computer Control of Undersea Teleoperators", MIT Man-Machine Systems Laboratory, 1978.
- R. Parasuraman, T. B. Sheridan and C. D. Wickens, "A Model for Types and Levels of Human Interaction with Automation", IEEE Transactions on Systems, Man, and Cybernetics 30(3), 2000, 286–297.
- M. R. Endsley and E. O. Kiris, "The Out-of-the-Loop Performance Problem and Level of Control in Automation", Human Factors 37(2), 1995, 381–394.
- L. J. Skitka, K. L. Mosier and M. Burdick, "Does automation bias decision-making?", International Journal of Human-Computer Studies 51(5), 1999, 991–1006.
- N. H. Mackworth, "The Breakdown of Vigilance during Prolonged Visual Search", Quarterly Journal of Experimental Psychology 1(1), 1948, 6–21.
- G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro and D. S. Weld, "Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance", CHI 2021.
- S. Breznitz, Cry Wolf: The Psychology of False Alarms, Lawrence Erlbaum, 1984.
- B. Green, "The Flaws of Policies Requiring Human Oversight of Government Algorithms", Computer Law & Security Review 45, 2022.
- tapqr.live, published product pillars including "AI Fraud Detection". Recorded as A25 in
SOURCE_INVENTORY.md, with the AI layer documented as future work as B9. https://tapqr.live
Related reading in this archive
- Designing with AI in the interface: the interface guarantees a model breaks, and how to restore them
- What AI-native product management actually means
- Latency, trust and the two variables everyone conflates
- TapQR: the product whose fraud-detection pillar is discussed above
- Agentic workflows as distributed transactions: the failure taxonomy behind the checkpoint argument