Dwarkesh Patel hosted a podcast with three AI researchers—Beren Millidge (CTO at Zyphra), John Schulman (chief scientist at Thinking Machines, former OpenAI co-founder who led RLHF work), and Charlie O'Neill (head of model training at Baseten)—to discuss why we might not see superintelligence by 2036. The conversation centers on technical bottlenecks rather than regulatory ones. Millidge argues the most likely scenario is a persistent sim-to-real gap that prevents models from generalizing beyond benchmarks, despite their apparent capability. Schulman points out that each new model release creates a cycle where people declare "this is AGI," but the model feels dumb within a month as users discover its actual limitations. He suggests we keep hitting bottlenecks in research and engineering where models can't fully replace human judgment or self-check effectively, preventing the explosive capability growth people expect. O'Neill makes the sharpest case: the current transformer-plus-RL recipe may be far from the global optimum of what a learner could theoretically achieve. He draws a parallel to Moore's Law, which looked like a straight line but required discrete innovations—like moving from pre-training scaling laws to RL—to keep climbing. If scaling the current paradigm hits diminishing returns and requires a fundamental discontinuity (throwing out gradient descent or neural networks entirely), then LLMs running at scale won't necessarily discover it. An LLM can't invent what's too far outside its training distribution, no matter how many copies you run. The group wrestles with whether the next breakthrough will be harder than anything since 2012. O'Neill distinguishes between incremental improvements layered onto the current approach versus finding something genuinely new. If we need the latter and it's truly distant from our current methods, scaling won't solve it. The underlying tension: if humans eventually would have discovered the next architecture, and LLMs can't, then maybe we're not on the path to AGI through current methods—we're just getting better at a local optimum.