Large machine learning models have advanced rapidly, with examples like Kimi-K2.5 featuring a staggering 1 trillion parameters. As these models grow, so does their demand for memory, computing power, and energy. Low-bit inference techniques are crucial for managing these demands, as they enable models to run faster and more cost-effectively by reducing the memory and compute requirements during user interactions. At Dropbox, products like Dropbox Dash rely on these techniques to efficiently handle vast amounts of user content, balancing model efficiency with hardware utilization and latency. Matrix multiplications are key to the performance of attention-based models, which are widely used for processing text, images, and audio. Specialized hardware on GPUs, such as NVIDIA’s Tensor Cores and AMD’s Matrix Cores, optimizes these operations significantly compared to traditional cores. Lowering numerical precision through quantization, which reduces the number of bits used to represent values (e.g., from 16-bit to 8-bit), enhances speed, memory efficiency, and energy consumption. For instance, FP4 support in Blackwell GPUs leads to notable energy savings compared to the previous H100 models. Quantization isn't one-size-fits-all; it encompasses various techniques that affect model accuracy and performance based on hardware capabilities. With diverse AI workloads, Dropbox must choose quantization formats carefully to balance latency and throughput. The recent introduction of the MXFP microscaling format has standardized low-bit data types with native support, dividing quantization into pre-MXFP and MXFP formats. Pre-MXFP formats often rely on software-managed scaling and integer data types, while MXFP formats integrate these operations directly into Tensor Core hardware, offering better performance for real-world inference tasks.