Pinterest engineered Manas, an embedding retrieval system handling billions of vectors with approximate nearest neighbor search, hybrid queries, and real-time updates. They optimized it through quantization techniques that cut computing resources by 10-15% while maintaining search accuracy.

• SIMD intrinsics on 8-bit integers reduced per-query compute by 10-15% without sacrificing performance
• Product quantization on disk storage paired with full-precision centroids improved both accuracy and throughput, diverging from the original SPANN approach
• The system supports hybrid queries combining token and embedding search with sub-second indexing of new content

ZeroModels is a library that ports 100+ model families to pure Keras 3, letting you run the same code on JAX, PyTorch, or TensorFlow without needing transformers or torch at runtime. Models load from preconverted weights, upstream checkpoints converted on-the-fly, or any Hugging Face repo, with a consistent two-call interface (build model, run prediction).

• 100+ model families (detectors, depth estimators, LLMs) work with identical code across three backends—switch backends by setting one environment variable before importing Keras
• Large models load with aggressive quantization built-in: GPT-OSS 120B stays at 66GB using bfloat16 + packed MXFP4 experts instead of 130GB fp32, with int8/int4/fp8 quantization available as arguments
• All outputs on model pages are measured from actually running the code snippets, not hand-written examples, so results are reproducible

Laguna XS 2.1 is a 33B Mixture-of-Experts model with 3B activated parameters per token, tuned for agentic coding and long-horizon tasks on local machines. It outperforms XS.2 on SWE-bench Multilingual by 5.4 points, adds support for major runtimes, offers FP8/INT4/NVFP4 checkpoints, and runs with a permissive OpenMDW-1.1 license. You can access it via API, OpenRouter, or locally with quantized or DFlash-boosted builds.

• Laguna XS 2.1 jumps from 57.7% to 63.1% on SWE-bench Multilingual, beating Qwen 3.6 (35B) and North Mini Code (30B) despite only 3B active params per token
• DFlash speculator models double token throughput versus stock XS 2.1 when run locally
• Same API pricing as XS.2 ($0.10 input/$0.20 output/$0.05 cache-read per million tokens), now served at 256K context
• XS.2 leaves the API in one week, pushing users to migrate to 2.1 (though it remains available on Baseten's Model Library)

This guide shows how to run Z.ai’s open-source GLM-5.2 model on local hardware using Unsloth Dynamic GGUF quantizations. It covers memory requirements for 1-bit and 2-bit setups, recommended inference settings, and step-by-step instructions for Unsloth Studio and llama.cpp. The article also explains KLD benchmarks and quantization accuracy trade-offs.

• GLM-5.2's 744B-parameter model can run on a 24GB GPU plus 256GB RAM using Unsloth's 2-bit dynamic quant (239GB, ~82% top-1 accuracy), an 84% size cut from the full 1.5TB model.
• The 1-bit quant shrinks total memory needs to ~223GB but drops accuracy to ~76.2%.
• 4-bit and 5-bit dynamic quants are nearly lossless per KLD testing, making them the better choice when accuracy on out-of-distribution tasks matters most.
• Dynamic quantization keeps critical layers at higher precision while aggressively compressing less important ones, which is why even 1-2 bit versions stay broadly usable despite the accuracy trade-off.

This article breaks down how an LLM turns your prompt into streamed tokens, covering tokenization, embeddings, transformer attention, and the two-phase pipeline of compute-bound prefill and memory-bound decode. It explains KV caching, quantization, and metrics like Time to First Token and Inter-Token Latency to show why inference speed depends on both compute and memory.

• LLM inference splits into compute-bound prefill (TTFT) and memory-bound decode (ITL), which explains why the two phases have fundamentally different performance bottlenecks.
• KV caching avoids quadratic recomputation but costs real memory—about 1 MB per token for a 13B model, so a 4,000-token context burns 4 GB of VRAM.
• Techniques like INT8/INT4 cache quantization, token eviction, and paged cache management exist specifically to tame this KV cache memory growth.
• DeepSeek's V4 cuts KV cache size by 90% at a million-token context by combining sparse and dense compressed attention.

PrismML’s Bonsai 8B trains a large language model with 1-bit weights from scratch, squeezing 8.2 billion parameters into just 1.15 GB. In benchmarks it ties or outperforms FP16 models like Llama 3.1 and runs at real-time speeds on phones, shifting the size-performance trade-off.

• Bonsai 8B packs 8.2B parameters into 1.15GB using native 1-bit weights, yet scores 70.5 average vs Llama 3.1's 67.1 (16GB FP16), even hitting 88.0 on GSM8K vs Llama's 76.6.
• It runs at 44 tokens/sec on an iPhone 17 Pro Max, making full on-device 8B-scale LLMs feasible without cloud infrastructure.
• Its "intelligence density" (score/GB) hits 1.062 versus Qwen's 0.098 and Llama's 0.084 — over 10x more capability per byte.
• Trade-offs appear in code generation (57.9 vs Qwen's 79.9 on HumanEval+) and multi-step reasoning (MuSR: 64.3 vs 70.0), showing 1-bit precision struggles with complex logical chains.

This article walks through why and how to run large language models locally, covering privacy, cost, offline access, and control. It breaks down hardware needs, quantization, PC versus Mac setups, and starter software to get models up and running.

• Local open-source models (GLM-5.1, Kimi K2.6) now match or beat top cloud models like Claude Opus 4.6 on coding benchmarks, handling 70-80% of typical daily AI tasks.
• 4-bit quantization (Q4_K_M) shrinks memory needs to ~0.6-0.7 GB per billion parameters with no noticeable quality loss.
• RTX 4090 PCs outperform M3 Max Macs on speed (100-140 vs 40-60 tokens/sec) for sub-24GB models, but Mac Studios with unified memory (up to 512GB) can run massive 671B-parameter models entirely in RAM.
• A 3090 PC build costs $1,500-2,000 and draws 450W with heavy noise, versus a quieter Mac Studio pulling only ~60W while supporting far larger models.