Click any tag below to further narrow down your results
Links
AirLLM is a framework that streams model layers from disk one at a time during inference, letting you run massive language models like DeepSeek-V3 (671B parameters) on single consumer GPUs without quantization or distillation. It also supports fine-tuning huge models on small VRAM by keeping only adapters in memory while streaming frozen weights.
- Runs a 671B parameter model on ~12GB VRAM and 125B models on ~6GB by keeping only one decoder layer on GPU at a time, with the rest streamed from disk
- Works with virtually every major open-source model family (Llama, Qwen, DeepSeek, Mistral, Gemma, etc.) through a single `AutoModel.from_pretrained()` call with no special configuration
- Supports optional 4-bit/8-bit weight quantization for up to 3x speedup, and fine-tuning of 125B models under 6GB VRAM by streaming base weights while keeping adapters resident
Phonon-2 is a 164 MB speech recognition model that matches the accuracy of a much larger teacher model while being 15 times smaller. It averages 5.21% word error across seven English test sets and runs extremely fast—transcribing an hour of audio in 20 seconds on a MacBook Air.
- Achieves 5.21% average word error rate across Open ASR Leaderboard benchmarks, beating larger models including Whisper large-v3-turbo (6.58%) and matching its 2.5 GB teacher model's accuracy despite being 1/15th the size
- Extreme speed: processes audio 174x faster than realtime on M5 MacBook Air, 143x on Zen 5 CPU, and 6,680x on H100 GPU in batches
- Quantized to 2.1 bits per weight using five learned levels, enabling the tiny download size while maintaining teacher-model performance
Liquid AI released Pipette, an open-source benchmarking suite for measuring AI model performance on phones, laptops, and embedded devices, plus DSpark draft models that speed up inference by 2-3x through speculative decoding. Both tools come with public datasets and are designed to let developers test the exact hardware-model-runtime combinations they plan to ship.
- Pipette provides a standardized way to benchmark 35+ model classes across different quantization levels, runtimes, and devices (phones, laptops, PCs, embedded hardware), with 10k+ verified results already in the warehouse and an interactive dashboard to compare configurations.
- DSpark draft models achieve 2.27x to 3.18x throughput gains on everything from M4 Max MacBooks to H100 GPUs by using lightweight draft models to propose token blocks that a target model verifies in one pass, without changing output quality.
- Both projects are open-source with native iOS and Android apps, public datasets, and submission workflows that let developers contribute benchmarks for their own hardware and model combinations.
ZeroModels is a library that ports 100+ model families to pure Keras 3, letting you run the same code on JAX, PyTorch, or TensorFlow without needing transformers or torch at runtime. Models load from preconverted weights, upstream checkpoints converted on-the-fly, or any Hugging Face repo, with a consistent two-call interface (build model, run prediction).
- 100+ model families (detectors, depth estimators, LLMs) work with identical code across three backends—switch backends by setting one environment variable before importing Keras
- Large models load with aggressive quantization built-in: GPT-OSS 120B stays at 66GB using bfloat16 + packed MXFP4 experts instead of 130GB fp32, with int8/int4/fp8 quantization available as arguments
- All outputs on model pages are measured from actually running the code snippets, not hand-written examples, so results are reproducible
Cohere released North Small Translate, a sparse mixture-of-experts model with 25 billion active parameters designed for machine translation across 50 languages. The model scores 83.60 on WMT26 benchmarks and is available on Hugging Face under a non-commercial license.
- The model uses a sparse MoE architecture with 218B total parameters but only activates 25B per token, reducing computational overhead compared to dense models of similar capability
- It supports 50 languages including major ones like English, Spanish, Mandarin, and Hindi, plus less common ones like Filipino and Maltese
- Multiple quantization options (BF16, FP8, NVFP4) let you run it on different hardware—from 8 H100s down to 2 H100s depending on precision, with production deployment using greedy decoding
Meta open-sourced Muse Glimmer, a 30-billion-parameter model designed to run on consumer GPUs and laptops for local AI agent tasks like scheduling, coding, and tool use. The model uses quantization and speculative decoding to fit within 20-32GB of memory while maintaining speed for real-time interaction. It's available now under Apache 2.0 license with integrations for llama.cpp, MLX, and other frameworks.
- Meta open-sourced Muse Glimmer, a 30B-parameter agentic model quantized to under 20GB so it runs locally on consumer GPUs/laptops (24-32GB) while competing with Gemma 4-31B and Qwen 3.6-27B.
- Speculative decoding with a lightweight drafter model proposes token blocks at once instead of one token at a time, speeding up reasoning and tool calls without changing outputs.
- Training combined logit distillation from a larger teacher model, agent-heavy mid-training data, and post-training RL/distillation across reasoning, coding, and agentic tasks.
- Released under Apache 2.0 with upcoming llama.cpp, MLX, and ExecuTorch integrations and support from Ollama, LM Studio, Together AI, and chipmakers like NVIDIA, Intel, AMD, and Arm.
The article breaks down which AI models and setups you can afford to run or train at home by 2026, comparing GPU costs, power use, and performance. It highlights efficient small-scale models, quantization tricks, and DIY hardware options to save money without sacrificing too much accuracy.
- Nvidia Blackwell cards should hit 150–200 TFLOPS FP16 under $1,500, making home rigs viable for large-model inference by 2026.
- 4-bit quantization plus FlashAttention already lets a 4090 run Llama 2-70B for under $0.02/inference and fit 13B models in 20GB VRAM, undercutting cloud A100 rental costs ($0.10–0.50/min).
- LoRA fine-tuning a 7B model on 8x4090s or two Blackwells takes a few hours and under $10 in electricity, though full 70B training from scratch still needs real clusters.
- A $3,000–4,000 home setup (with ~$50–100/month power costs for 24/7 use) will be enough to prototype LLM applications without cloud fees.
Laguna XS 2.1 is a 33B Mixture-of-Experts model with 3B activated parameters per token, tuned for agentic coding and long-horizon tasks on local machines. It outperforms XS.2 on SWE-bench Multilingual by 5.4 points, adds support for major runtimes, offers FP8/INT4/NVFP4 checkpoints, and runs with a permissive OpenMDW-1.1 license. You can access it via API, OpenRouter, or locally with quantized or DFlash-boosted builds.
- Laguna XS 2.1 jumps from 57.7% to 63.1% on SWE-bench Multilingual, beating Qwen 3.6 (35B) and North Mini Code (30B) despite only 3B active params per token
- DFlash speculator models double token throughput versus stock XS 2.1 when run locally
- Same API pricing as XS.2 ($0.10 input/$0.20 output/$0.05 cache-read per million tokens), now served at 256K context
- XS.2 leaves the API in one week, pushing users to migrate to 2.1 (though it remains available on Baseten's Model Library)
This guide shows how to run Z.ai’s open-source GLM-5.2 model on local hardware using Unsloth Dynamic GGUF quantizations. It covers memory requirements for 1-bit and 2-bit setups, recommended inference settings, and step-by-step instructions for Unsloth Studio and llama.cpp. The article also explains KLD benchmarks and quantization accuracy trade-offs.
- GLM-5.2's 744B-parameter model can run on a 24GB GPU plus 256GB RAM using Unsloth's 2-bit dynamic quant (239GB, ~82% top-1 accuracy), an 84% size cut from the full 1.5TB model.
- The 1-bit quant shrinks total memory needs to ~223GB but drops accuracy to ~76.2%.
- 4-bit and 5-bit dynamic quants are nearly lossless per KLD testing, making them the better choice when accuracy on out-of-distribution tasks matters most.
- Dynamic quantization keeps critical layers at higher precision while aggressively compressing less important ones, which is why even 1-2 bit versions stay broadly usable despite the accuracy trade-off.
Whichllm is a single-command CLI that detects your GPU/CPU/RAM, pulls live benchmarks from HuggingFace, and ranks the best fitting local LLMs by real performance metrics. It also lets you simulate different GPUs, generate Python snippets, run chats, output JSON, and plan hardware upgrades.
- One command (`uvx whichllm@latest`) detects your GPU/CPU/RAM and ranks local LLMs by live, aggregated benchmarks (LiveBench, Artificial Analysis, Aider, Chatbot Arena, Open LLM Leaderboard) instead of just parameter count.
- Scores are confidence-weighted by data freshness/source (direct vs. self-reported vs. interpolated) and adjusted for runtime fit—VRAM overhead, quantization discounts, and MoE active-vs-total parameter splits.
- It can simulate hypothetical GPUs, reverse-plan what hardware you'd need for a specific model, and output JSON for scripting or piping into jq.
- Example rankings: RTX 4090/3090 tops out with Qwen3.6-27B Q5_K_M (score 92.8, ~27 t/s), 8GB VRAM cards get Qwen3-14B Q3_K_M (score 71.0, ~22 t/s), and CPU-only setups land a 20B MoE model at ~6 t/s (score 45.2).
@akshay_pachaar on X
A first-principles walk through what happens between your prompt and the streamed reply: tokenization, attention, and why the model does two very different jobs with two different bottlenecks. The piece argues that KV cache memory, not raw model intelligence, is what usually limits long-context speed.
- Generation is a loop that predicts one token at a time, and the first token is slower than the rest because prefill is compute-bound while decode is memory-bound.
- The KV cache avoids recomputing attention over the whole sequence, but at roughly 1 MB per token for a 13B model, a 4K context burns about 4 GB of VRAM.
- Quantization is the highest-leverage knob: a 7B model drops from 14 GB at FP16 to 3.5 GB at INT4, often with only a point of benchmark loss.
- DeepSeek's V4 previews reportedly cut the cache to about 10% of its predecessor's size at a one-million-token context, showing the field is redesigning attention around the cache.
This article walks through why and how to run large language models locally, covering privacy, cost, offline access, and control. It breaks down hardware needs, quantization, PC versus Mac setups, and starter software to get models up and running.
- Local open-source models (GLM-5.1, Kimi K2.6) now match or beat top cloud models like Claude Opus 4.6 on coding benchmarks, handling 70-80% of typical daily AI tasks.
- 4-bit quantization (Q4_K_M) shrinks memory needs to ~0.6-0.7 GB per billion parameters with no noticeable quality loss.
- RTX 4090 PCs outperform M3 Max Macs on speed (100-140 vs 40-60 tokens/sec) for sub-24GB models, but Mac Studios with unified memory (up to 512GB) can run massive 671B-parameter models entirely in RAM.
- A 3090 PC build costs $1,500-2,000 and draws 450W with heavy noise, versus a quieter Mac Studio pulling only ~60W while supporting far larger models.