CohereLabs/North-Small-Translate-1.0 · Hugging Face
Cohere released North Small Translate, a sparse mixture-of-experts model with 25 billion active parameters designed for machine translation across 50 languages. The model scores 83.60 on WMT26 benchmarks and is available on Hugging Face under a non-commercial license.
- The model uses a sparse MoE architecture with 218B total parameters but only activates 25B per token, reducing computational overhead compared to dense models of similar capability
- It supports 50 languages including major ones like English, Spanish, Mandarin, and Hindi, plus less common ones like Filipino and Maltese
- Multiple quantization options (BF16, FP8, NVFP4) let you run it on different hardware—from 8 H100s down to 2 H100s depending on precision, with production deployment using greedy decoding
Cohere Labs released North Small Translate, a sparse Mixture-of-Experts model built for machine translation across 50 languages. The model has 25 billion active parameters out of 218 billion total, meaning most weights stay dormant unless needed—a design choice that keeps inference faster than running the full network. It supports a 16K context window for both input and output, and Cohere offers three quantization levels (BF16, FP8, and 4-bit) depending on your hardware. The smallest version runs on a single Nvidia B200 GPU with 4-bit weights, while the full precision version needs 8 H100s. The model uses a CC BY-NC license but requires accepting Cohere's Acceptable Use Policy, and you need to sign up to download the weights.
Using the model requires some care with memory management and output parsing. If you load it with PyTorch's `device_map="auto"`, you need to explicitly set `max_memory` to leave room for temporary buffers when fusing the Mixture-of-Experts layers—otherwise you'll run out of VRAM partway through loading. The model wraps its responses in structural markers like `<|START_TEXT|>` and `<|END_TEXT|>` that aren't registered as special tokens, so `skip_special_tokens=True` won't remove them. Cohere provides a parsing library called melody that strips these markers cleanly, which works with both direct transformers calls and vLLM deployments.
The documentation covers three main ways to run it: directly via transformers with `AutoModelForCausalLM`, through the transformers pipeline abstraction, or via vLLM for serving. You can override the default system instructions by passing `platform_instruction_override` to the chat template, or append additional instructions as a regular system message. Cohere recommends greedy decoding (no sampling) for production use, which matches how they serve the model themselves. The vLLM setup requires installing melody 0.9.0+ and configuring tensor parallelism based on your GPU count—for H100s, they suggest `tp 8` for the full model.