Gemma 4 models, recently launched by Google DeepMind, consist of four variations with distinct parameters: E2B (2 billion), E4B (4 billion), 31B (31 billion), and a Mixture of Experts model (26B A4B) that activates 4 billion parameters during inference. These models are designed to be multimodal, processing text, images, and audio. The architecture builds on Gemma 3 but introduces key enhancements, particularly in attention mechanisms. One major change is the interleaving of local and global attention layers. Each model maintains a fixed pattern, with smaller models using a 4:1 ratio of local to global layers, while larger ones implement a 5:1 pattern. This design improves computation efficiency by limiting the number of tokens processed at one time. For instance, the smaller E2B and E4B models have a sliding window of 512 tokens, while the larger models utilize a 1024-token window. This approach allows them to maintain context without overwhelming computational resources. Gemma 4 employs several techniques to optimize global attention layers, which are typically resource-intensive. Grouped Query Attention (GQA) is used, allowing multiple query heads to share key-value pairs, thus reducing memory requirements. The K=V trick further minimizes memory use by ensuring that keys and values are equivalent in the global attention layers. Lastly, the p-RoPE method enhances positional encoding, providing a way for the models to track word order effectively through a rotational mechanism that varies in frequency, aiding in capturing the nuances of language.