The author chronicles their experience in training a domain-specific language model using a 1 billion parameter Llama 3-style architecture on 8 NVIDIA H100 GPUs. They plan to establish a basic pre-training infrastructure, using Karpathy's fine-web-edu-shuffled dataset for training. The author opts for a 2048 token sequence length, acknowledging the limitations of context length during training due to quadratic attention costs. They also highlight the importance of optimizing memory usage and provide a detailed breakdown of expected memory requirements during various stages of the training process. A significant challenge arises with memory allocation during training. The author encounters discrepancies between memory usage estimates on different hardware, particularly between their Mac and the H100 GPUs. They discover that FP32 inputs on H100 GPUs lead to higher memory consumption due to the fallback to naive attention implementations. This issue prompts the author to explore automatic precision lowering for specific operations to optimize memory usage further. The article outlines various optimization strategies, including using `torch.compile` to improve performance. The author calculates a need for 20 billion tokens for effective training, targeting a batch size of 1 million tokens, resulting in around 200,000 training steps. They plan to implement additional techniques like gradient accumulation and mixed precision training to enhance efficiency. The focus is on balancing memory usage and computation to maximize training throughput while minimizing overhead.