Click any tag below to further narrow down your results
Links
This is NVIDIA's model card for Nemotron-3.5-Bolt-E4B-4GB, a small 4B-effective model post-trained from Gemma-4-E4B-it and aimed at fast, low-memory local voice and device assistants. The page is mostly deployment instructions, covering how to run it through llama.cpp, vLLM, SGLang, Ollama, LM Studio and several agent frameworks.
- Built for low memory and low time-to-first-token, with improved multi-turn tool use and instruction following over its Gemma base.
- Ships as a Q4_K_M quantized GGUF with up to 64K tokens of context, and supports toggling reasoning mode through the chat template.
- An optional DFlash drafter model enables speculative decoding for faster generation, and the model is licensed under OpenMDW-1.1 for commercial or non-commercial use.
This is the Hugging Face model card for TwIL-8B, an 8.8B-parameter model fine-tuned from IBM's Granite 4.2 for math, code and instruction following. The card covers how to run it through Transformers, llama.cpp, vLLM, Ollama and several local apps, plus benchmark results against similar-sized open models.
- Scores 98.2 on MATH-500 and 81.6 on LiveCodeBench (verbose variant), the top results in its comparison group, with a Q4_K_M quantized GGUF about 5 GB so it runs on modest hardware.
- Trails on knowledge-heavy benchmarks: GPQA Diamond is 11.5 points behind Qwen3.5-9B, and the model is not a general knowledge model.
- Uses about 38% fewer output tokens per AIME 2026 problem than Qwen3.5-9B, though it is still far more verbose than frontier models.
Backburner splits a 27-billion-parameter language model between a MacBook Pro and iPhone connected via USB-C, achieving 29-44% faster inference on long contexts by distributing computation across both devices' GPUs. The system maintains identical outputs to running the model on the Mac alone while supporting up to 196k tokens of context on an iPhone 17 Pro Max.
- Split prefill workload: Mac runs layers 1-40 while iPhone runs layers 41-64 in parallel, with the iPhone holding older KV cache pages past 64k context and computing attention over them during both prefill and decoding.
- Measured performance gains: 44% faster throughput at 16k context (157 vs 109 tok/s), and ability to maintain 8-bit precision at 128k tokens versus the Mac alone dropping to 4-bit to fit in memory.
- Greedy decoding is token-identical between Mac-only and Mac+iPhone configurations across tested context lengths (8k, 32k, 140k), confirming the split computation produces correct results.
This article argues that Ollama’s custom backend runs 30–70% slower than llama.cpp, locked users’ models in a proprietary format, and dropped proper attribution before grudgingly reverting. It also criticizes Ollama Cloud’s reliability and warns that the platform has strayed from its local-first, open-source roots.
- Ollama's custom ggml fork runs 30-70% slower than llama.cpp (e.g., 30 vs 52 tokens/sec on RTX 5090 with Qwen3 Coder 32B), despite reverting to upstream llama.cpp in v0.30.0-rc15
- Ollama locked models into a proprietary hashed-filename registry format between 2024 and early 2026, preventing reuse with other tools
- Ollama Cloud Pro users report up to 95% failure rates, long timeouts, broken tool calling, and unanswered support tickets, undermining its local-first pitch
- Ollama mislabeled distilled DeepSeek-R1 variants as the full 671B model and launched its desktop app without source code despite open-source claims