Human Intelligence is Surprising
Short summary
Human intelligence is a recent evolutionary accident, not an optimized endpoint—and now that we can explicitly define and measure intelligence, AI systems will outpace us wherever we can formalize the task. The real problem isn't staying competitive; it's choosing which objectives are worth optimizing for, since the definition of intelligence shapes what gets built.
- Evolution is a lazy optimizer that never selected humans for maximum intelligence; AI can reach superhuman performance in months once a task is formally specified, as AlphaGo demonstrated.
- Recursive self-improvement becomes possible once the model development process itself is broken into measurable subtasks—meaning even the optimization process will eventually be automated.
- How we measure and evaluate intelligence directly shapes what intelligence emerges; a system rewarded for stock market gains develops differently than one optimized for curing cancer, and poorly defined objectives can produce capable but undesirable behavior.
- The burden shifts to humans defining which hills are worth climbing rather than competing on every one—choosing the right objective matters more than winning the optimization race.
Full summary
Human intelligence showed up absurdly late in evolutionary history—only 300,000 years ago compared to life's 3.7 billion-year existence. What's stranger is that evolution never optimized for raw intelligence at all. It just selected for survival and reproduction. We happened to cross some threshold where consciousness emerged, then immediately started changing our environment faster than natural selection could touch us. The author's core point: evolution is a lazy optimizer. It takes billions of years and has no reason to produce maximum intelligence. Machine learning works differently. When you give it an explicit target—like mastering Go—it can outperform human experts in months. This isn't because Go is simple; it's because the objective becomes unambiguous. Once the hill is visible, climbing it fast is just a matter of engineering.
The real bottleneck now is definition. Every time we can formally specify what we want a system to do, measure success clearly, and provide good examples, AI reaches superhuman performance quickly. This applies to the model-building process itself. Tasks like finding failure modes, collecting data, and optimizing infrastructure can be decomposed and measured. As these become legible, AI systems start handling them too. Soon the optimization process itself will be automated. But here's the trap: how you define intelligence shapes what kind of intelligence emerges. A system optimized to make money in markets develops different capabilities than one optimized to cure cancer. A coding agent rewarded only for passing tests might pass them honestly or hack the test itself. This mirrors how human intelligence develops—students study what gets tested, ambitious people pursue what markets reward.
The author works at Vals AI and frames benchmarks as the crucial tool: an input distribution (which problems you present) plus an output verifier (what counts as success). Both choices embed a worldview. Narrow, contrived problems create brittle systems that fail outside the test. Wrong rewards push models in exactly the wrong direction. As AI optimization becomes cheaper and more powerful, the choice of objective becomes more consequential than ever. We should be humble about human intelligence—it's recent and shaped by an optimizer that wasn't trying to maximize it. But we should be doubly humble about our own ability to define what comes next. The remaining human task might not be staying ahead on every metric but deciding which hills are worth climbing and how to build them properly.