The Preference Cascade Is Only Getting Started
Short summary
A cascade of high-profile resignations and media coverage has shifted public and elite opinion on AI existential risk, with nearly two-thirds of Americans now saying AI poses at least moderate risk of destroying humanity. The article argues this momentum is real but insufficient—actual technical and policy work on AI safety must accelerate to match the growing concern.
- Public opinion has shifted dramatically: 65% of Americans now see moderate-to-high risk of AI destroying humanity (up ~15% from previous polling), and 93% of top executives disagreed with Trump's characterization of AI dangers as exaggerated.
- AI researchers themselves estimate an ~18% median chance of AI causing human extinction, with Asian researchers estimating even higher risks than Western counterparts.
- The preference cascade alone won't solve the problem—the article warns that without actual technical progress on alignment and safety, plus regulatory action like export controls and transparency rules, public concern will prove meaningless.
Full summary
A major shift in public discourse about AI existential risk is underway. An AI Impacts survey shows researchers estimate an 18% chance of AI causing human extinction—nearly 1 in 5. This has translated into measurable public opinion change: recent Politico polling found nearly two-thirds of Americans now see at least moderate risk of AI destroying humanity, up roughly 15 percentage points from previous surveys. The cascade started when Leopold Coxon quit Anthropic and posted about safety concerns, which went viral because enough Americans had finally developed context on AI to care, and influential accounts amplified it quickly. Even business leaders are on board: 93% of executives at a Yale event disagreed with Trump's claim that AI dangers are exaggerated.
The author argues this preference cascade is necessary but insufficient. Real change requires solving underlying technical problems, not just pacing development. Key figures like Matthew Yglesias are now stepping up to explain why safety-focused policy isn't doom-mongering—it's basic risk management, like fixing a leak before it gets worse. Yglesias proposes a sequenced approach: start with light-touch transparency rules, enforce aggressive export controls, then negotiate from a position of strength before moving to tougher regulations. The author worries there may not be time for this gradual strategy if timelines are shorter than it assumes.
The cascade has reached mainstream media and elite institutions. Op-eds in the New York Times, Free Press, and Wired are making the case that AI could genuinely kill us all. Stephen Witt describes a "vibe shift" where researchers recognize AI may be slipping from human control, offering four policy options ranging from shutting down research entirely to maintaining a kill switch. Meanwhile, opponents—particularly Nvidia and a16z—are mounting political resistance, planting hit pieces against safety-focused organizations. The author notes there's a conference, AGI.WTF, happening September 22-23 at Lighthaven to continue building momentum.