This article discusses new methods for enhancing the efficiency of large language models through sparsity. It examines various strategies like relufication and error budget thresholding to achieve significant speedups in on-device inference while maintaining accuracy. The authors are developing a unified framework in PyTorch to streamline these techniques.

Generative AI, particularly Large Language Models (LLMs), is much cheaper to operate than commonly believed, with costs decreasing significantly in recent years. A comparison of LLM pricing to web search APIs shows that LLMs can be an order of magnitude less expensive, challenging misconceptions about their operational costs and sustainability. The article aims to clarify these points for those who hold the opposite view.