Benchmarks are central to showcasing AI model performance, yet they are frequently misunderstood. Every few weeks, a new model claims to surpass state-of-the-art results, often accompanied by misleading bar charts that suggest a straightforward increase in intelligence. Understanding how these benchmarks function is key to interpreting these claims. A benchmark score results from a combination of the model, its settings, the harness used for testing, and the scoring criteria. Changing any part of this equation can significantly alter the outcome, making it essential to analyze the entire setup rather than focusing solely on the model name. The benchmarking process is fraught with inconsistencies. For example, the code used to evaluate models is often flawed, leading to incorrect results due to bugs or overly strict criteria. Variability is another issue; even with controlled settings, running the same model multiple times can yield different scores. Labs often report metrics in ways that obscure true performance differences, and modifications to benchmark code can skew results. Additionally, the model evaluated may differ from the one users interact with in production, leading to discrepancies in performance. Factors like latency and cost rarely accompany benchmark scores, leaving out critical considerations for real-world applications. The article also critiques specific benchmarks, such as LMArena, where users vote on model responses. While it captures user preferences, it can suffer from saturation and may not reliably rank models. Misleading scores and variability make it challenging to gauge true model performance. A deeper examination of these benchmarks reveals numerous pitfalls and highlights the need for a critical approach when interpreting their results.