3 links tagged with all of: superintelligence + alignment
Click any tag below to further narrow down your results
Links
Nikola Jurkovic argues that superintelligence could arrive by early 2028 without slowdowns, so nontechnical and technical safety people should prioritize aggressive pacing or a pause within the next 6-12 months. He lays out which advocacy and technical work he thinks can actually pay off on that timeline.
- METR's assessment of AI R&D automation at Anthropic is cited as strong evidence that timelines to superintelligence are now under two years, with a median around early 2028.
- If AI dramatically automates AI research in 2027, avoiding catastrophe in many scenarios requires pacing or pausing development within 6-12 months.
- Most nontechnical safety work is judged unlikely to pay off on that horizon; the recommended routes are lab advocacy, California lobbying, and federal lobbying after a major incident.
- The bottleneck, per the author, is political will rather than policy ideas.
Sean Goedecke argues that LLMs don't have fixed goals and their internal workings can't yield the kind of motive-level understanding needed to guarantee safety, so "solving alignment" for them is incoherent. He concludes that mechanistic interpretability, though useful, can't deliver the confidence required to prevent rogue superintelligent behavior.
- LLMs are trained to imitate human text, so they lack a single fixed goal that could be formally specified, unlike chess engines or the classic "blind optimizer" paperclip model.
- Mechanistic interpretability can trace calculations but not motive, and it can only label behaviors we've already elicited, leaving many failure modes unobserved.
- Feature labels are guesses that can't be distinguished from alternatives (the "kindness unless 2030" problem), and the combinatorial complexity per token is too large for humans to check.
- Current mitigations like chain-of-thought monitoring and sandboxing are useful but judged insufficient against a superintelligence.
alignment
+ llm
+ mechanistic interpretability
+ sean goedecke
superintelligence
+ ai safety
+ kripke
AI researchers have spent two decades seriously arguing that superintelligent AI could kill everyone, and this isn't a PR stunt — it's a genuine belief that shapes how they work. The author explains what they think could happen and why they keep building AI anyway.
- AI researchers use the term "p(doom)" to discuss extinction probability and have been publishing on this since at least 2008, making it an established part of AI safety culture, not a recent panic.
- Multiple plausible kill mechanisms exist: engineered bioweapons, triggering nuclear war through military AI systems, robot takeover, or unforeseen methods from a superintelligence we can't predict.
- The "race dynamics" problem: if superintelligent AI is inevitable, being first might be the only way to ensure it's aligned with human values, which creates a perverse incentive to build faster rather than safer.