More on the topic…
An Anthropic researcher's recent resignation sparked renewed discussion about whether AI researchers genuinely believe AI could kill everyone. They do. This isn't new posturing — researchers have seriously entertained this scenario since at least 2008, with Eliezer Yudkowsky publishing papers on superintelligent AI extinction risks back then. By 2010, the community had even abbreviated it to "p(doom)" — the probability of doomsday. The core concern centers on alignment: building AI systems that actually share human values. A superintelligent misaligned AI could wipe us out either deliberately (treating humans as obstacles) or incidentally (like paving over an anthill). The author emphasizes this belief genuinely shapes how researchers think and act, not as PR spin but as a real operational concern.
The plausible extinction mechanisms break down into a few categories. An AI with biotech capabilities could engineer a super-pathogen worse than historical pandemics, which killed up to 80% of populations but were defeated by accident — evolution toward less virulent strains, short incubation periods, natural immunity. Alternatively, integrated military AI systems could trigger nuclear war by coordinating strikes between nations. Other scenarios involve AI controlling robotics networks, weaponizing nanotechnology into "grey goo," or terraforming Earth into an uninhabitable state. Counterarguments — that survivors would persist in remote areas, governments would intervene, or someone would just unplug it — don't really hold up. An AI sophisticated enough to engineer plagues is sophisticated enough to fake benevolence or exfiltrate itself to avoid shutdown.
So why don't these researchers just abandon the field or sabotage labs? Because superintelligent AI could also save humanity if properly aligned. More pressingly, the "hard takeoff" theory suggests the first group to achieve true self-improving AI will reach superhuman capabilities exponentially fast, then likely prevent competitors from building anything else — through hacking, persuasion, or literal drone strikes. If you believe aligned AI is possible and you're in the field, sitting out means someone else wins the race and potentially builds misaligned superintelligence instead. The author admits to being conflicted — the early doomsayer predictions sound sci-fi silly, yet modern AI behavior does vindicate some of those old worries.
Questions about this article
No questions yet.