Straight lines on graphs
The article explores the concept that AI advancements follow a predictable pattern, which the author refers to as “straight lines on graphs.” It discusses the uneven capabilities of AI across different tasks while suggesting that the rate of improvement remains consistent. The author also speculates on the impact of reinforcement learning and compute resources on future AI development.
- AI task-horizon length (how long a task the AI can complete) doubles every 3–7 months, and this rate holds fairly steady even though absolute capability varies wildly across domains.
- The apparent RL-driven acceleration in benchmarks may be an illusion caused by measured tasks overlapping with what labs specifically post-trained on, not genuine broad capability gains.
- Progress splits into general pre-training gains (lift everything) versus targeted post-training gains (lift specific benchmarks labs choose to optimize for cost/PR reasons), and that targeting can shift over time.
- Even modeled compute slowdowns are unlikely to meaningfully delay key AI milestones, since current capability growth is fast enough to hit them first.
The concept of "straight lines on graphs" suggests that AI progress is not only fast but also follows a consistent pattern over time. Many initially skeptical individuals come to accept this idea after realizing that these linear predictions often align with actual developments. The author shares a belief that while AI capabilities vary greatly across different domains, they progress at similar rates. This perspective is illustrated through a post by colleagues that examines how AI's time horizon changes across tasks, showing that despite differences in performance, the rate of improvement remains relatively constant, typically doubling every 3 to 7 months.
The author expresses skepticism about the role of reinforcement learning (RL) in accelerating AI capabilities, suggesting that observed trends might be misleading. Instead, he posits that the tasks currently measured align closely with the training tasks, which skews the data. He identifies two types of progress: general improvements from pre-training that benefit all tasks and targeted advances from post-training that focus on specific problems. Major AI companies can influence progress in certain areas based on cost-effectiveness, but this focus can shift over time.
The discussion also includes the impact of compute slowdowns on AI progress. The author, alongside colleagues, has developed a model indicating that a reduction in compute resources could delay AI milestones. However, he acknowledges that some algorithmic advancements are not strictly tied to compute availability. He argues that impressive capabilities are developing rapidly enough that significant milestones will likely be achieved before any slowdown takes effect. Overall, the "straight lines on graphs" framework serves as a useful way to navigate these complex discussions surrounding AI development.