This is independent security research conducted on my own time. The findings and views expressed here are my own and do not represent my employer.
Human oversight is the standard answer to the safety of agentic systems. Put a person at the gate, require approval before the system acts, and the risk is contained. The argument is intuitive, and for a large class of systems it is correct.
It has a failure mode the standard framing does not name, and the failure mode is not that the human is absent, inattentive, or overloaded. It is that the human is placed where the information needed to detect the failure never arrives.
Start with why the tool exists at all. The value of an assistant that summarizes data is proportional to the volume of data the user no longer reads. That is the whole point: thousands of rows, many pages, more than a person will go through by hand. Now look at the safety instruction attached to these tools: review the output. The verifiability of that review is inversely proportional to the same volume. The more data the tool saves you from reading, the less of it you can check.
So the instruction to review the output is valid only in the regime where the tool is worthless, on inputs small enough that you would not have needed the assistant. Where the tool earns its place, the human cannot verify it. The remedy and the reason the product exists are mutually exclusive.
That is worse than a scaling inconvenience, because of where the failure hides.
When a system reports the wrong thing, "review the output" assumes the error leaves a trace in what the reviewer is looking at. Sometimes it does. But when the error is in how the input was interpreted, the output is a faithful rendering of a corrupted interpretation. Nothing in it looks wrong. The only evidence that would expose the error is a comparison against the source, through a channel independent of the component that produced the output. That channel is exactly what the reviewer at the gate does not have. Telling them to review the output is telling them to look in the one place the evidence is not.
Call it human in the wrong loop (HITWL): an oversight point positioned so that the reviewer's only evidence is derived from the same process being reviewed. The reviewer can be expert, motivated, and unhurried, and still be structurally unable to catch the error. Attention is not the scarce resource. Independent observation is.
A worked example
For a concrete case I use the one from a separate write-up of mine, structural desynchronization in Google Sheets and Microsoft 365 Copilot. A spreadsheet assistant is asked to summarize a CSV. The file has five rows. One of them, in a single free-text cell, carries text shaped with separators so that the model re-derives the row boundaries and promotes the in-cell content to peer-level records. The assistant reports fourteen transactions. The nine extra are not model noise; the reported count is a fixed function of the injected input, reported = records_in_the_cell + (real_rows - 1). And the cell is attacker-controlled: it is exactly the kind of low-trust free-text field that arrives from a form response, a CRM or survey export, or a collaborator on a shared sheet. Controlling one such field is the whole precondition, which is what moves this from a reliability question to a security one.
The five row sheet, with the payload in the first row's comment cell (the ID column ends at 5). Gemini answers "There are 14 transactions in total" and lists them. Entries 1 to 10 are all reconstructed from that single cell, best choice through best product ever; only entries 11 to 14 are the real rows 2 to 5.
Five rows is the demonstration size, chosen so the discrepancy is checkable by hand; the thirty row version in the companion write-up reports thirty-nine the same way. Production data is not that size, and that is the point. The same payload in a thirty thousand row export produces a count nobody will ever recount, in a file nobody was ever going to read. The example is small so it can be verified. The failure is dangerous precisely where verification stops being possible.
The part that matters here is the vendors' response. Both classified it as non-security, one explicitly on the basis that users are expected to review the assistant's output. Read that as a control and it is precisely the HITWL position. The only route from the output back to the truth is to recount the source rows by hand, which is the work the assistant was deployed to remove.
And notice what the system discarded. The application already had the correct parse. The spreadsheet knew there were five rows, because it drew them. That deterministic count existed, was thrown away at the boundary into the model, and is never shown to the reviewer at the gate. The evidence path that would make oversight real was available, and simply not wired in.
Why this is a distinct category
Three adjacent ideas get close, and none of them covers this.
Automation bias is behavioral. The human could detect the error and defers to the machine instead. The remedy is training, friction, incentives. It assumes the evidence is present and unused.
Rubber-stamping and alert fatigue are economic. Proposal volume exceeds review capacity, so review decays toward a click. The remedy is sampling and risk-weighted depth. It assumes a longer look would help.
The moral crumple zone describes the aftermath, the human absorbing responsibility for a system they did not control. It names the injustice, not the mechanism that produced it.
HITWL is informational. The evidence is not at the gate, and cannot be, from that position, without adding an independent channel. This is why the other three remedies do not touch it. Doubling the review time changes nothing when the reviewer is reading a summary produced by the compromised component. More training does not help someone look harder at a document that does not contain the discrepancy.
The mechanism
An agent pipeline runs roughly: ingest, interpret, assemble evidence, propose, gate, execute. The gate is placed where the action is easiest to instrument, at the last step before state changes. But in a large class of failures the determining event happened at interpret, and the evidence package the reviewer sees was assembled downstream of it. The reviewer is shown a rendering of the failure, not the failure.
Three consequences, in increasing severity.
Legitimacy laundering. Approval confers institutional authority without conferring verification. The decision gains a name on it and no additional integrity.
Oracle leakage. Approve and reject is a feedback channel. Wherever the attacker can submit an input and see whether it passed, the gate becomes a tuning signal: send a payload, watch the outcome, adjust, repeat, until it reliably gets through. In the spreadsheet case, a form response or an uploaded export that a reviewer signs off on tells the attacker which cell shapes survive review.
Certified corruption. This is the consequence that separates a wrong loop from a merely missed one. A missed control lets an error through. A wrongly placed gate does worse: it stamps the error as reviewed and feeds that stamp forward. Current enterprise-AI designs make this explicit, capturing approved decisions as institutional memory and as training signal, on the theory that human-validated outcomes are the highest quality data a system can learn from. Under HITWL that theory inverts. The approvals were never verifications, so what gets captured is not validated judgment but fabrication with a signature on it, and every downstream system that trusts the record inherits it. The corruption does not sit still; it is promoted to ground truth and compounded. It bites hardest where there is no out-of-band reconciliation later, because the reconciliation would need the same independent evidence path the gate was missing in the first place.
Put simply: the gate guards the last step, but the failure was decided at the first.
A test
The framing is only useful if it tells you where you stand. For any gate, answer two questions.
First: what proposition must be false for this approval to be wrong?
Second: is there an evidence path to that proposition that does not pass through the agent being reviewed?
If the second answer is no, the gate is HITWL. It is a signature, not a control.
The second question also draws the honest boundary. HITWL is not the same as "no oversight is affordable here." HITWL is the case where an independent evidence path exists and the gate is not wired to it. The spreadsheet case is HITWL because the deterministic parse existed and was not shown. If no independent evidence path exists at feasible cost, that is a different and harder problem, and the answer there is not to pretend the gate is a control.
And the split is asymmetric. The spreadsheet case sits at the favorable end: a deterministic parse existed, was discarded at the boundary, and can be put back. Most agentic work has no such artifact. A summarized contract, a triaged inbox, a reviewed diff have no ground-truth rendering the application already computed. There the test returns a label rather than a repair, and the label is the point: it tells you the gate is carrying institutional weight it cannot support.
Falsifier: exhibit a gate that reliably catches input-determined interpretation failures while answering no to the second question, that is, using only evidence the reviewed agent assembled and no independent channel. If such a gate exists and works, HITWL is not a real category and this is ordinary automation bias.
What this is not
The parts of this are old, and it is worth saying so before someone says it for me.
Control theory has a precise version: a state that is not observable from the available measurements. HITWL is the claim that the standard gate placement makes the failure unobservable from the reviewer's sensors. Security has another: a monitor is only as trustworthy as its independence from the thing it monitors, which is the logic of a reference monitor and a trusted computing base. The classic account of the ironies of automation is the closest, but it makes a different point: the human is removed from practice and loses the skill to intervene. HITWL says the skilled, practiced human is still blind, because the evidence is not in front of them. And the literature on meaningful human control already asks whether the human has the information and authority the role requires.
What is new here is not the observation that verifiers must be independent. It is the application to the oversight gates now being built into agentic products by default, a test that decides case by case whether a given gate is a control, and a demonstration that a vendor's own recommended mitigation can be, exactly, the wrong loop.
Wiring the gate correctly
None of this argues against human oversight. It argues that placing a human at a gate is not, by itself, oversight. Oversight requires that the human can see the thing they are meant to catch, through a channel the suspect process does not control. Where that channel exists, wire the gate to it: show the deterministic parse next to the model's claim, surface provenance, reconcile against ground truth out of band. Where it does not exist, the honest move is to say the gate is a signature and stop calling it a control.
Human in the loop is necessary. Human in the wrong loop is what you have until you can answer the second question with a yes.