@alexolegimas - Anthropic Study: Experts Defer

Open original

@alexolegimas on X A randomized trial by Anthropic with 50 professional data analysts found that experts follow an AI agent's advice most of the time, yet few of them credit the agent for the decision. The author argues this makes "human in the loop" oversight a weaker safeguard than people assume.

  • In a randomized trial with 50 professional data analysts, experts followed the agent's advice the vast majority of the time.
  • Only a tiny fraction of experts credited the agent; most believed they reached their decisions on their own.
  • If people defer to AI without noticing, human oversight of AI safety work may give less assurance than expected.

The core finding comes from a randomized trial Anthropic ran with 50 professional data analysts, led by Alex Tamkin and colleagues. The question was whether experts who hand work to an AI agent still make the key decisions themselves. The answer, at least in this setup, is mostly no. Analysts followed the agent's recommendations the large majority of the time in a causal sense, meaning the agent's advice was what actually moved their choices. Yet only a small fraction of them credited the agent. Most believed they had reached their conclusions on their own. The author, who flagged the post as one of the more important pieces of human-AI interaction research they've seen lately, draws a parallel to Nisbett and Wilson's classic work on people being shaped by outside influences they can't identify. That older literature showed people confidently explaining choices using reasons that had no real bearing on what they did, and often missing the actual causes entirely. The new study looks like a modern version of the same blind spot, now with an AI in the causal chain. Why does this matter? The post argues that judgment calls are where human values actually enter an analysis or a decision, and as more work, including AI safety research, gets routed through agents, those calls become the main lever people have. If people defer without noticing, then the common reassurance of "human in the loop" offers less protection than it sounds like. A person can be formally in control while their judgment has quietly been set by the system they're supposed to be supervising. That is the practical worry: oversight that exists on paper but not in the decision itself.

Ask about this article