This article breaks down how an LLM turns your prompt into streamed tokens, covering tokenization, embeddings, transformer attention, and the two-phase pipeline of compute-bound prefill and memory-bound decode. It explains KV caching, quantization, and metrics like Time to First Token and Inter-Token Latency to show why inference speed depends on both compute and memory.

PrismML’s Bonsai 8B trains a large language model with 1-bit weights from scratch, squeezing 8.2 billion parameters into just 1.15 GB. In benchmarks it ties or outperforms FP16 models like Llama 3.1 and runs at real-time speeds on phones, shifting the size-performance trade-off.

This article walks through why and how to run large language models locally, covering privacy, cost, offline access, and control. It breaks down hardware needs, quantization, PC versus Mac setups, and starter software to get models up and running.