AI researchers have spent two decades seriously arguing that superintelligent AI could kill everyone, and this isn't a PR stunt — it's a genuine belief that shapes how they work. The author explains what they think could happen and why they keep building AI anyway.

• AI researchers use the term "p(doom)" to discuss extinction probability and have been publishing on this since at least 2008, making it an established part of AI safety culture, not a recent panic.
• Multiple plausible kill mechanisms exist: engineered bioweapons, triggering nuclear war through military AI systems, robot takeover, or unforeseen methods from a superintelligence we can't predict.
• The "race dynamics" problem: if superintelligent AI is inevitable, being first might be the only way to ensure it's aligned with human values, which creates a perverse incentive to build faster rather than safer.

The article argues that “AGI” has become a fuzzy umbrella term with conflicting definitions, making it useless for tracking AI progress or predicting major shifts. It urges us to drop AGI in favor of clear, concrete milestones—like fully automated AI R&D, human-level adaptability, or self-sufficient AI—to ground discussions and forecasts.

• "AGI" now spans wildly different thresholds, from matching human test performance to running civilization unaided, making claims like "AGI arrived in early 2025" vs "it's a decade off" talk past each other
• Uneven, jumpy progress across tasks means definitional gaps will keep widening rather than converging
• Concrete milestones (fully automated AI R&D, human-level adaptability, self-sufficient AI, machine consciousness) or vivid benchmarks (10x annual energy growth, interstellar probes) are proposed as replacements for fuzzy labels like AGI or superintelligence
• Swapping vague umbrella terms for specific milestones would make safety, regulation, and timeline discussions more reliable

This article discusses the need for new industrial policies to manage the transition to superintelligent AI. It emphasizes the importance of democratic processes in shaping AI's future, ensuring broad access and mitigating risks. The authors argue for proactive measures to ensure that AI benefits everyone and addresses potential disruptions to jobs and society.

• OpenAI is calling for explicit industrial policy to manage the transition to superintelligent AI rather than leaving it to market forces alone
• The company frames democratic governance and broad access as necessary to prevent AI benefits and power from concentrating among a small few
• Past technological transitions show that translating growth into widespread prosperity required deliberate political action, not automatic diffusion, and superintelligence will require similarly ambitious intervention
• The proposal centers on building new institutions and safeguards to keep advanced AI systems controllable and aligned while rethinking the social contract around jobs and economic participation