Letter 109: All About Local LLMs
Short summary
This article walks through why and how to run large language models locally, covering privacy, cost, offline access, and control. It breaks down hardware needs, quantization, PC versus Mac setups, and starter software to get models up and running.
- Local open-source models (GLM-5.1, Kimi K2.6) now match or beat top cloud models like Claude Opus 4.6 on coding benchmarks, handling 70-80% of typical daily AI tasks.
- 4-bit quantization (Q4_K_M) shrinks memory needs to ~0.6-0.7 GB per billion parameters with no noticeable quality loss.
- RTX 4090 PCs outperform M3 Max Macs on speed (100-140 vs 40-60 tokens/sec) for sub-24GB models, but Mac Studios with unified memory (up to 512GB) can run massive 671B-parameter models entirely in RAM.
- A 3090 PC build costs $1,500-2,000 and draws 450W with heavy noise, versus a quieter Mac Studio pulling only ~60W while supporting far larger models.
Full summary
Local LLMs have closed the gap with cloud services in speed and accuracy. In early 2026, GLM-5.1 outperformed Claude Opus 4.6 on a coding benchmark, and Kimi K2.6 just took the lead. These open-source models run on your own machine—no server round-trip, no per-token fees, and full data privacy. For day-to-day tasks like drafting, summarization, or simple coding, a local model handles about 70–80% of what you’d throw at a top-tier cloud AI.
Five reasons to run LLMs locally: privacy, cost, no rate limits, offline access, and total control. Your prompts stay on your hard drive, inference has zero marginal cost once you buy the hardware, and you never worry about credit burn or unexpected service blocks. With a downloaded model you can work on flights, in remote areas, or behind a corporate firewall. You choose quantization levels—4-bit models (Q4_K_M) cut memory needs to roughly 0.6–0.7 GB per billion parameters without noticeable quality loss.
Hardware matters most by memory pool size. On a PC, an NVIDIA RTX 4090 handles 8 B-parameter models at 100–140 tokens/sec, beating Apple’s M3 Max (40–60 tokens/sec) for models under 24 GB. But Macs with unified memory shine on giant models: a Mac Studio with 512 GB can run a 671 B-parameter model entirely in-RAM. PCs cost around $1,500–2,000 for a 3090 build, draw 450 W under load and roar like a jet engine. A Mac Studio is quieter, pulls ~60 W, and can host models tens of times larger without offloading data.