This article critiques the use of perplexity as a metric for evaluating machine learning models, particularly Transformers. It argues that a model can achieve low perplexity while failing to predict certain sequences accurately, highlighting the metric's inadequacy in reliably selecting the best model. The authors provide analytical insights into how model confidence and accuracy relate to perplexity.