AI researchers debate how close we are to recursive self-improvement

Open original

Three AI researchers from frontier labs discuss why transformative AI might fail to emerge despite current progress—exploring technical bottlenecks like the sim-to-real gap, the possibility that current methods hit diminishing returns, and whether scaling alone can discover the next paradigm shift needed for AGI.

  • Current AI progress follows repeating cycles where new models seem revolutionary but reveal limitations within months, potentially continuing indefinitely without reaching true generalization or self-improvement loops.
  • The transformer + RL paradigm may require fundamental discontinuities to advance further, similar to how scaling laws in deep learning repeatedly hit walls that required new innovations (pre-training, then RL) to overcome—and future breakthroughs might not be discoverable by scaling current methods alone.
  • Even if AI systems become better than humans at research, the jump to explosive recursive self-improvement isn't guaranteed; there could be persistent gaps between what works in simulation and real-world deployment that prevent the takeoff scenario.

Dwarkesh Patel hosted a podcast with three AI researchers—Beren Millidge (CTO at Zyphra), John Schulman (chief scientist at Thinking Machines, former OpenAI co-founder who led RLHF work), and Charlie O'Neill (head of model training at Baseten)—to discuss why we might not see superintelligence by 2036. The conversation centers on technical bottlenecks rather than regulatory ones. Millidge argues the most likely scenario is a persistent sim-to-real gap that prevents models from generalizing beyond benchmarks, despite their apparent capability. Schulman points out that each new model release creates a cycle where people declare "this is AGI," but the model feels dumb within a month as users discover its actual limitations. He suggests we keep hitting bottlenecks in research and engineering where models can't fully replace human judgment or self-check effectively, preventing the explosive capability growth people expect. O'Neill makes the sharpest case: the current transformer-plus-RL recipe may be far from the global optimum of what a learner could theoretically achieve. He draws a parallel to Moore's Law, which looked like a straight line but required discrete innovations—like moving from pre-training scaling laws to RL—to keep climbing. If scaling the current paradigm hits diminishing returns and requires a fundamental discontinuity (throwing out gradient descent or neural networks entirely), then LLMs running at scale won't necessarily discover it. An LLM can't invent what's too far outside its training distribution, no matter how many copies you run. The group wrestles with whether the next breakthrough will be harder than anything since 2012. O'Neill distinguishes between incremental improvements layered onto the current approach versus finding something genuinely new. If we need the latter and it's truly distant from our current methods, scaling won't solve it. The underlying tension: if humans eventually would have discovered the next architecture, and LLMs can't, then maybe we're not on the path to AGI through current methods—we're just getting better at a local optimum.

Ask about this article