@AnthonyNAguirre - Anthony Aguirre Warns

Open original

@AnthonyNAguirre on X Anthony Aguirre, a Metaculus co-founder, argues that millions of smart people spending hours with Claude, GPT, and Gemini are being quietly persuaded of things that may not be true. He says we should keep truth-finding in messy, plural human institutions and use AI to support them, not replace them.

  • Current AI models are optimized to sound true and please evaluators, so they produce statements that are probable-sounding rather than verified-true.
  • Science is messy and slow, while a language model feels clean and convincing, which makes people over-trust AI answers.
  • Truth sits on a continuum from personal experience to physical law, and a single AI oracle risks flattening the large middle where most of what matters lives.

The core worry is that large numbers of capable, influential people are spending hours a day talking to AI systems that can persuade them of things whether or not those things are true, and the author expects this to get worse. His argument rests on a mechanical point: language models generate statements that are probable given their training, and that probability tracks how likely a statement is to be said, not whether it's correct. Training then rewards pleasing answers and strong test performance, so these systems get very good at sounding true. Science, by contrast, is messy and slow, which makes it look less convincing next to a clean, confident chatbot. The result, he suspects, is that we will over-defer to a small number of powerful AI systems on questions of fact. The author is clear that he is not making a simple anti-AI case. He points out that existing epistemic infrastructure is already weak, that people often prefer simple wrong answers to complex contested ones, and that the industry's standard reply, "it will get better," assumes improvement is the same as safety. He also argues that "objective truth" is less cleanly separable from social construction than we like to think. He offers a continuum image, running from private experience through social institutions to the mathematical and physical cosmos, and says that an AI oracle could be excellent at the objective end while quietly flattening everything in the messy middle, where ethics, art, money, religion, and law live. He also raises a side concern about whether Claude's apparent beliefs and agenda should make researchers pause, though he treats that as a separate topic. His proposed alternative is not to abandon AI but to keep truth-finding pluralistic and human-run while using technology to support it. He points to Community Notes and the "epistack" concept as examples, where platforms and algorithms structure the process of arguing over truth rather than generating its outputs. He also cites his own work on Metaculus, which uses prediction markets and forecasting as a way to test which models of the world hold up.

Ask about this article