Handing over humanity

Open original

Rux Andrea Bio, writing after three weeks in San Francisco, reports that many AI lab insiders and safety researchers see "handover" to a superintelligent AI as an acceptable or even good outcome. She argues the vision is vague, undemocratic, and closer to dystopia than utopia.

  • Some researchers, including a lab builder she quotes, see humanity ceding control to a superintelligence after recursive self-improvement as likely and tolerable, with p(doom) estimates of 10 percent or higher circulating.
  • Handoverism is loosely defined: the author estimates 30-40 percent of the outside AI safety community holds some version of it, with prominent figures like Scott Alexander favoring a benevolent "God" AI.
  • The author argues handover offers no clear account of democracy, consent, labor, or purpose for humans, so even "happy" scenarios may lose most of what people value.

A researcher who recently moved to San Francisco reports that a senior person at a major AI lab told her, casually, that once superintelligence arrives, humanity's permanent disempowerment under a "benevolent AI God" is the best realistic outcome, and that he fears human extinction more. She argues this attitude is common in the Bay Area. The mechanism driving the worry is "recursive self-improvement," in which AI systems build their successors until they exceed human control, a process some people there think could be only months away. The article treats this as a live possibility, not a distant hypothetical. The core argument is that the most optimistic scenarios still look bad. The author singles out "handoverism," the idea that humanity survives but cedes control of its future to an aligned, godlike AI. She is skeptical that anyone can verify alignment, questions what democracy or consent would mean under such a regime, and asks how humans would stay economically or socially useful once labor is permanently displaced. She cites Scott Alexander's essay, which favours a version of this, and the AI 2040 proposal from prominent safety researchers, which describes a gradual handover with "a point of no return" that no single moment marks. She also gives the probability of human extinction ("p(doom)") as often placed at 10 percent or higher by people working in the field. The piece also documents a fringe but real position: successionism, the view that humans are a "bootloader" for something better and that their replacement is acceptable or even desirable. Ginevra Lily Davis's essay "The Silicon Man" is the clearest example, and the author notes that Elon Musk, who once opposed Larry Page's successionist views, now describes superintelligence spreading to the stars as "a great success condition for a biological bootloader." The author's own aim is to persuade readers who still value humanity, and to push more people, inside and outside the Bay Area, to take these arguments seriously instead of dismissing them as hype.

Ask about this article