A Visual Guide to Gemma 4
This article introduces the Gemma 4 family of models from Google DeepMind, detailing their architectures and improvements over the previous version, Gemma 3. It highlights key features such as interleaved attention layers and efficiency enhancements in global attention mechanisms.
- Gemma 4 comes in four sizes—E2B, E4B, 31B, and a 26B MoE model that only activates 4B parameters at inference—and handles text, images, and audio
- Local-to-global attention layers are interleaved at a fixed 4:1 ratio in smaller models and 5:1 in larger ones, with sliding windows of 512 tokens (E2B/E4B) or 1024 tokens (larger models) to cut compute costs
- Global attention memory use is reduced through Grouped Query Attention, a K=V trick that makes keys and values identical, and p-RoPE for improved positional encoding
Gemma 4 models, recently launched by Google DeepMind, consist of four variations with distinct parameters: E2B (2 billion), E4B (4 billion), 31B (31 billion), and a Mixture of Experts model (26B A4B) that activates 4 billion parameters during inference. These models are designed to be multimodal, processing text, images, and audio. The architecture builds on Gemma 3 but introduces key enhancements, particularly in attention mechanisms.
One major change is the interleaving of local and global attention layers. Each model maintains a fixed pattern, with smaller models using a 4:1 ratio of local to global layers, while larger ones implement a 5:1 pattern. This design improves computation efficiency by limiting the number of tokens processed at one time. For instance, the smaller E2B and E4B models have a sliding window of 512 tokens, while the larger models utilize a 1024-token window. This approach allows them to maintain context without overwhelming computational resources.
Gemma 4 employs several techniques to optimize global attention layers, which are typically resource-intensive. Grouped Query Attention (GQA) is used, allowing multiple query heads to share key-value pairs, thus reducing memory requirements. The K=V trick further minimizes memory use by ensuring that keys and values are equivalent in the global attention layers. Lastly, the p-RoPE method enhances positional encoding, providing a way for the models to track word order effectively through a rotational mechanism that varies in frequency, aiding in capturing the nuances of language.