nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB · Hugging Face
This is NVIDIA's model card for Nemotron-3.5-Bolt-E4B-4GB, a small 4B-effective model post-trained from Gemma-4-E4B-it and aimed at fast, low-memory local voice and device assistants. The page is mostly deployment instructions, covering how to run it through llama.cpp, vLLM, SGLang, Ollama, LM Studio and several agent frameworks.
- Built for low memory and low time-to-first-token, with improved multi-turn tool use and instruction following over its Gemma base.
- Ships as a Q4_K_M quantized GGUF with up to 64K tokens of context, and supports toggling reasoning mode through the chat template.
- An optional DFlash drafter model enables speculative decoding for faster generation, and the model is licensed under OpenMDW-1.1 for commercial or non-commercial use.
NVIDIA-Nemotron-3.5-Bolt-E4B-4GB is a small language model built by post-training Google's Gemma-4-E4B-it. The pitch is fast local use: low memory footprint and low time-to-first-token, aimed at voice assistants, interactive personas, and embedded or edge devices. The stated gains over the base Gemma are in multi-turn tool use, instruction following, and grounding responses in provided input. Listed specs are 8B total parameters with 4B effective, a Q4_K_M quantization, and a 64K-token context window for both input and output. Inputs and outputs are text only. Thinking mode and a conversational mode are toggled through the chat template, and the card recommends temperature 1.0 with top_p 0.95. A companion DFlash drafter is available for speculative decoding, and a full-precision BF16 variant exists under a separate repo. The license is described as permitting both commercial and non-commercial use.
Most of the page is a deployment catalog rather than model analysis. It lists how to run the GGUF quant through llama.cpp (install via brew, WinGet, or source build, then serve an OpenAI-compatible endpoint on port 9931 or run it in the terminal), vLLM and SGLang (pip or Docker, with a curl example against the chat completions route), Ollama, Docker Model Runner, LM Studio, Jan, Lemonade, and agent frameworks like Pi, Hermes Agent, and OpenClaw, each with config snippets pointing at the local server. The Transformers section shows standard pipeline and AutoModel loading. One detail worth noting: the llama.cpp deployment entry is marked "recipe pending," so the llama.cpp commands on the page may not be fully supported yet for this checkpoint.
The remaining sections are the model card proper, but the excerpt cuts off before the benchmark numbers, training data details, and evaluation results. What is visible covers architecture, training methodology headings, prompting guidelines, datasets, and ethical considerations, without the figures themselves. For anyone deciding whether this fits a low-latency on-device use case, the benchmarks and training-data sections are the parts to read in full before trusting the marketing framing of "improved" tool use and grounding.