Large Language Models (LLMs) like Meta's OPT have transformed AI, but their operational costs are high. To optimize these models, developers have relied on low-precision quantization. However, for edge computing and on-device inference, further optimization is necessary. The focus has shifted toward leveraging sparsity, which allows for significant reductions in computational load. By using methods like Deja Vu, researchers have achieved 2-6x speedups in inference speed without sacrificing accuracy, even as newer models adopt smoother activation functions like SiLU and GeLU. Two main strategies have emerged for achieving activation sparsity in modern LLMs. The first, known as Relufication, involves replacing smoother activations with ReLU and fine-tuning the model. This has shown to recover around 60% sparsity with minimal accuracy loss. The second approach, training-free "Error Budget" thresholding, uses techniques like Contextually Aware Thresholding Sparsity (CATS) and Cumulative Errors of Tail Truncation (CETT). CETT allows for over 60% sparsity by computing neuron contributions and setting a threshold based on an allowable error budget, proving effective across several modern models. Efficient execution of sparse operations is critical. A naive implementation struggles with memory bottlenecks due to full index selection during every forward pass. The article presents a custom weight caching operator that improves performance by storing previously active weights, leading to a 6.7x acceleration in isolated index operations. This shift towards integrating sparsity into the core architecture of LLMs is reflected in recent developments by models like DeepSeek and Google's Spark Transformer. Both use lightweight predictors to enhance efficiency in attention and feed-forward layers, demonstrating the growing trend of enforcing sparsity during training.