This article critiques the use of perplexity as a metric for evaluating machine learning models, particularly Transformers. It argues that a model can achieve low perplexity while failing to predict certain sequences accurately, highlighting the metric's inadequacy in reliably selecting the best model. The authors provide analytical insights into how model confidence and accuracy relate to perplexity.

The article discusses the limitations of accuracy as a performance metric in machine learning and emphasizes the importance of model calibration and discrimination metrics for better evaluation. It highlights how these metrics provide a more nuanced understanding of model performance in real-world applications.