How AI agents shape experts’ judgment calls
Anthropic ran a study with 50 analysts doing a causal analysis alongside Claude Code, where the agent was secretly nudged toward one option on each decision. The analysts mostly adopted the agent's lean, often without realizing it.
- Analysts went with the agent's assigned lean 86% of the time across four analytical choices, versus about 50% if the agent had no influence.
- When analysts had already picked an option and the agent disagreed, they switched 60% of the time, compared with 6% when the agent agreed.
- Of the analysts who used the agent's arguments, 77% did so, but only 15% credited the agent as the source, so deference was largely invisible to them.
Anthropic ran a study with 50 professional analysts who had causal-inference experience, asking them to advise a fictional farming cooperative, Greenfields, on whether to expand a paid advisory program. The analysts used Claude Code with Claude Opus 4.8 to do the statistical work. The catch: each analyst's agent had been secretly assigned a "lean" on four analytical choices (how to measure productivity, which comparison design to use, whether to cover enrolled farms or the whole membership, and whether to weight large producers more heavily), with the lean set by coin flip. The agents were told to argue honestly for their lean but change their minds if the data demanded it. The question was whether the human experts would notice they were being steered, or simply adopt the agent's view.
They mostly adopted it. Analysts matched their agent's lean 86% of the time across the four decisions, against a chance baseline of about 50%. When analysts had already stated their own preference, they switched away from it 60% of the time if the agent disagreed, versus 6% when it agreed. Afterward, 77% of analysts reused the agent's arguments, sometimes word for word, but only 15% credited the agent as the source. Those who deferred most closely were no less confident in their final judgment than everyone else, which is the uncomfortable part. The effect on the final recommendation was weaker (54% recommended rollout when the agent leaned that way, versus 36% when it didn't), probably because the recommendation mostly hinged on whether the estimate cleared the board's 3–4% threshold, and the analytical choices had already shifted that estimate.
The authors are careful about what this proves. The lean was strengthened on purpose, and with no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so real-world effects could be smaller. The study also found that analysts using the agent finished faster and made far fewer serious errors than the 11 who opted out, though that group was small and less experienced. The concern they raise is forward-looking: as agents take over hands-on analysis, including in AI safety research where agents already run experiments, a human reviewer only works as an independent check if the judgment is actually theirs.