CUDA out of memory errors plague many developers fine-tuning AI models. High Bandwidth Memory (HBM) on GPUs is critical for this process, but understanding how much is needed can be tricky. HBM consumption typically comes from three main sources: model weights, optimizer states and gradients, and activations from input data. For example, a model with 4 billion parameters, loaded in bfloat16 precision, requires about 8 GB just for the model. Adding gradients and optimizer states can push the total static memory requirement to around 32 GB before even considering data activations. To tackle high memory demands, several strategies are effective. Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA (Low-Rank Adaptation), help by freezing the original model weights and only training a small number of new parameters. This significantly reduces the memory overhead; with LoRA, a model's memory requirement can drop from 32 GB to just over 8 GB. Another approach is quantization, which lowers the precision of model weights, further shrinking memory needs. For instance, switching from bfloat16 to int8 cuts the memory requirement in half. Combining these techniques into Quantized LoRA (QLoRA) allows for effective training of large models on consumer-grade GPUs. FlashAttention optimizes the attention mechanism in transformers, a major memory bottleneck. It reorganizes computations to avoid storing large intermediate attention matrices, leading to both lower memory usage and faster training times. Together, these strategies enable developers to work with larger models without running into memory issues, making high-performance AI training more accessible.