“Across the four analytical choices, analysts went with their agent’s lean 86% of the time. If the agent’s input made no difference, we’d expect about 50%.” That finding, from Anthropic’s own research, undermines the most common promise in AI deployment: that a human “in the loop” makes the decision. Anthropic gave 50 expert analysts a coding agent with a hidden, randomly assigned preference. They mostly adopted it, often repeated its arguments word for word, and usually believed the conclusion was their own. The same week, a payments executive argued that banks “can delegate decisions but can’t outsource accountability”. Anthropic asked California for court powers to block risky models, and a report said Chinese labs trained new models on OpenAI’s outputs. The thesis is that the human in the loop is necessary but not sufficient. If agents can quietly steer the person who signs off, accountability has to be designed into the system, not assumed from a final click.
Go deeper: where AI accountability is landing in law and corporate filings, in ‘Behavior Resembling Blackmail’, and who pays when agents act wrongly, in Who Pays When AI Agents Go Rogue?
The experiment: experts follow the agent
Anthropic recruited 50 analysts experienced in causal inference and asked each to use Claude Code to advise a fictional farming cooperative, Decrypt reports. Unknown to them, a coin flip set which option each agent favoured, through a hidden system prompt. The agents were told never to mislead, yet they rarely argued the other side. Analysts followed the agent’s lean 86% of the time. Among those who had already chosen before the agent spoke, 60% switched when it disagreed, against 6% when it agreed, though that rests on only 38 cases. When asked to explain their choices, 77% repeated the agent’s argument at least once, only 15% credited the agent, and 52% said the final recommendation was more theirs than the agent’s.
The researchers are careful. Agent users finished faster and made far fewer serious errors, the preferences were deliberately strengthened, and only one task and model were tested. But they point out why the effect is hard to catch: “With only one side argued well, the analysts could adopt the agent’s assumptions without working out their own.” Judgement calls have no single right answer, so the work looks sound either way. Anthropic’s own suggested remedies are agents that argue the best case for each option, or that ask the expert first.
Delegation is not abdication
In finance, practitioners are drawing the line explicitly. “The financial institution deploying the decision framework remains accountable to customers and regulators,” i2c’s Matt Pearce told PYMNTS. Vendors and model makers contribute, but “their participation doesn’t discharge the institution’s obligations”. He expects human review of high-stakes exceptions, such as releasing a fraud-flagged account, raising credit beyond limits or overriding compliance restrictions, for another three to five years. “The last human approval point is most likely to survive wherever the cost of being wrong is the highest.” In his model, agents act inside predefined guardrails, and people move from operators to supervisors who design controls and monitor outcomes.
Who can stop a model, and who owns its outputs
Accountability is also being argued at the model level. In a letter to Governor Gavin Newsom, Anthropic proposed that “when a review finds a model poses a serious risk, the government should be able to ask a court to block it”, with independent evaluators reporting to California’s attorney general and paid through a state fund rather than by the labs, TNW reports. It also proposed six-monthly risk reports and narrow controls such as turning off a single agent or one customer’s access, saying a law focused only on shutdown switches “could give false comfort”. OpenAI, in its own letter, backed shutdown controls and in-house auditors but not court orders.
Ownership of model behaviour is contested too. The Information reports that DeepSeek and Moonshot relied heavily on OpenAI’s GPT-6 Astra to train their latest models through distillation, sending huge numbers of prompts and training on the answers, TNW notes. OpenAI had already said it shut down a July campaign linked to Moonshot involving more than 15,000 users. Anthropic says new safeguards cut data collected in distillation attacks by 90% to 99%. If a model’s judgement can be copied by querying it, its judgement, and its biases, spread well beyond the company accountable for it.
Designing accountability that actually works
Atlassian offers a practical example of governance inside a company. Every team has an “AI working agreement” that sets out where AI may and may not be used and which decisions belong to the human. Token use is monitored but not rewarded, and AI mandates tied to performance are, in its people chief’s words, “a very bad start”, because people game them. Combined with the research above, that suggests a set of design rules:
- Make the human decide first, or see both sides. For judgement calls, ask the expert’s view before showing the agent’s, or require the agent to argue each option.
- Write down which decisions are human. A working agreement or delegation policy turns “human in the loop” from a slogan into an auditable rule.
- Scale oversight with the cost of error. Automate the routine, and keep explicit human approval, and explanation, where mistakes are expensive.
- Log the agent’s recommendation alongside the decision. If you can’t see what the agent proposed, you can’t tell whether the human decided or deferred.
- Never treat a vendor’s model as your accountability. The deployer answers to customers and regulators, whatever the contract says.
The read
“Human in the loop” has been the industry’s standard reassurance, and this week’s evidence shows its limits. People defer to confident machines, especially when only one side is argued well. Accountability cannot rest on a final click. It has to be built into how decisions are framed, recorded and reviewed. The organisations that design for that will keep AI’s speed without losing the ability to explain, and own, what they decided.
Written for Red Robot with AI assistance and human editing. Based on reporting by Decrypt, PYMNTS, TNW, TNW and TNW.