Quinn Slack discusses a new metric called "Off-the-Rails Cost," which compares the performance of AI models Sonnet, Gemini, and Opus. He highlights that 17.8% of costs for Gemini users are tied to "wasted threads," significantly worse than the other models. This analysis aims to improve Amp's functionality and may lead to automatic detection of these issues.

This article breaks down how AI benchmarks work and highlights their limitations. It discusses factors influencing benchmark results, such as model settings and scoring methods, and critiques common practices that can distort performance claims.