I Tested the 1-Bit LLM That Fits in 1 GB — It Shouldn't Be This Good | by Chew Loong Nian | in Level Up Coding

Open original

PrismML’s Bonsai 8B trains a large language model with 1-bit weights from scratch, squeezing 8.2 billion parameters into just 1.15 GB. In benchmarks it ties or outperforms FP16 models like Llama 3.1 and runs at real-time speeds on phones, shifting the size-performance trade-off.

  • Bonsai 8B packs 8.2B parameters into 1.15GB using native 1-bit weights, yet scores 70.5 average vs Llama 3.1's 67.1 (16GB FP16), even hitting 88.0 on GSM8K vs Llama's 76.6.
  • It runs at 44 tokens/sec on an iPhone 17 Pro Max, making full on-device 8B-scale LLMs feasible without cloud infrastructure.
  • Its "intelligence density" (score/GB) hits 1.062 versus Qwen's 0.098 and Llama's 0.084 — over 10x more capability per byte.
  • Trade-offs appear in code generation (57.9 vs Qwen's 79.9 on HumanEval+) and multi-step reasoning (MuSR: 64.3 vs 70.0), showing 1-bit precision struggles with complex logical chains.

PrismML’s Bonsai 8B squeezes 8.2 billion parameters into just 1.15 GB by training with native 1-bit weights instead of the usual 16- or 32-bit floats. In head-to-head tests against Llama 3.1 8B, Qwen 3 8B and other 8 billion-parameter models (all around 16 GB in FP16), Bonsai hit a 70.5 average score across six benchmarks—beating Llama 3.1 (67.1) and LFM2 8B (69.6), and nearly matching Mistral 3 8B (71.0). It even posted an 88.0 on GSM8K math, outperforming Llama’s 76.6, despite fitting on a phone and running at 44 tokens per second on an iPhone 17 Pro Max. PrismML coins “intelligence density” as score per gigabyte. Bonsai’s 1.062 score/GB dwarfs Qwen’s 0.098 and Llama’s 0.084, underscoring how much model capability they pack into each byte. The trade-offs show up in code generation (57.9 vs Qwen’s 79.9 on HumanEval+) and non-English tasks, where community tests reveal weaker performance. Multi-step reasoning also dips (MuSR: 64.3 vs Qwen’s 70.0), suggesting the 1-bit precision cap hits when tasks demand tight logical chains or complex syntax. Deploying Bonsai is straightforward via PrismML’s demo repo, a llama.cpp fork or Ollama. The GGUF file is 1.16 GB and loads in seconds on Apple Silicon or CUDA GPUs. This shift makes true on-device LLMs feasible—an 8 B model on a Raspberry Pi or smartphone, no cloud needed. Lower memory translates to 14× more users per GPU or a move to cheaper hardware, slashing hosting costs. Local inference also sidesteps data-leak risks, a boon for privacy-sensitive fields. Other variants (4 B and 1.7 B) promise even leaner footprints for embedded use cases.

Ask about this article