GitHub - lyogavin/airllm: AirLLM 70B inference with single 4GB GPU
Short summary
AirLLM is a framework that streams model layers from disk one at a time during inference, letting you run massive language models like DeepSeek-V3 (671B parameters) on single consumer GPUs without quantization or distillation. It also supports fine-tuning huge models on small VRAM by keeping only adapters in memory while streaming frozen weights.
- Runs a 671B parameter model on ~12GB VRAM and 125B models on ~6GB by keeping only one decoder layer on GPU at a time, with the rest streamed from disk
- Works with virtually every major open-source model family (Llama, Qwen, DeepSeek, Mistral, Gemma, etc.) through a single `AutoModel.from_pretrained()` call with no special configuration
- Supports optional 4-bit/8-bit weight quantization for up to 3x speedup, and fine-tuning of 125B models under 6GB VRAM by streaming base weights while keeping adapters resident
Full summary
AirLLM is a Python library that streams model layers from disk during inference, letting you run enormous language models on GPUs with far less VRAM than normally required. The core trick is breaking models into individual layers and loading them one at a time as needed, rather than keeping everything in memory. This means you can run a 70B parameter model on a single 4GB GPU without quantization, distillation, or pruning. The recent updates show genuinely wild scaling: Kimi K3 (2.8 trillion parameters) fits in 3.72GB, Qwen3.8-Flash-Next (125B) needs 5.95GB, and even DeepSeek-V3 (671B) runs on ~12GB. The library also recently added training support, letting you fine-tune these massive models on modest hardware by streaming frozen weights and keeping only adapters in VRAM.
The technical approach focuses on the actual bottleneck rather than brute-force optimization. Since loading from disk is the limiting factor, not computation, the developers only quantize weights (not activations), which is simpler and preserves accuracy better than full quantization. They offer 4-bit and 8-bit block-wise compression that can speed things up by 3x. The library handles model decomposition automatically—it splits your downloaded model into layer files on first run, which requires disk space but then stays there for reuse. You can optionally delete the original downloaded model to reclaim half the disk space.
Usage is straightforward: install via pip, then use `AutoModel.from_pretrained()` with a Hugging Face model ID, and inference works exactly like standard transformers code. The library supports most popular models—Llama, Qwen, DeepSeek, ChatGLM, Mistral, and others—with an auto-detection system so you don't need to specify model classes. It runs on Linux, macOS (Apple silicon only), and even supports CPU inference. The main trade-off is speed: layers stream from disk during generation, so inference is slower than having everything in VRAM, but the ability to run 671B models on consumer hardware makes that worthwhile for many use cases.