Thread by @liquidai on Thread Reader App
Short summary
Liquid AI released Pipette, an open-source benchmarking suite for measuring AI model performance on phones, laptops, and embedded devices, plus DSpark draft models that speed up inference by 2-3x through speculative decoding. Both tools come with public datasets and are designed to let developers test the exact hardware-model-runtime combinations they plan to ship.
- Pipette provides a standardized way to benchmark 35+ model classes across different quantization levels, runtimes, and devices (phones, laptops, PCs, embedded hardware), with 10k+ verified results already in the warehouse and an interactive dashboard to compare configurations.
- DSpark draft models achieve 2.27x to 3.18x throughput gains on everything from M4 Max MacBooks to H100 GPUs by using lightweight draft models to propose token blocks that a target model verifies in one pass, without changing output quality.
- Both projects are open-source with native iOS and Android apps, public datasets, and submission workflows that let developers contribute benchmarks for their own hardware and model combinations.
Full summary
Liquid AI and Artificial Analysis released Pipette, a benchmarking suite designed to measure how AI models actually perform on consumer devices—phones, laptops, desktops, and embedded hardware. The tool addresses a real gap: existing benchmarks focus on cloud-served models, but people increasingly want to run AI locally. Pipette lets you compare a model's quality, speed, latency, and memory use across different combinations of quantization levels, runtimes, and hardware in one place. The initial release covers 35 model classes with 7 quantization levels, tested on a MacBook Pro M4 Max, iPhone 17 Pro, Samsung Galaxy S26 Ultra, and an AMD Ryzen AI Max+. They've already compiled over 10,000 benchmark results across these configurations, with context lengths ranging from 256 to 8,192 tokens depending on device memory.
The release includes an interactive dashboard at pipette.liquid.ai where you can explore results, native benchmark apps for iOS and Android so you can test on your own hardware, and open-source infrastructure under Apache 2.0 licensing. The data is public, and they're inviting third-party submissions—if you want a specific model-device-runtime combo measured, you can run the benchmark clients yourself and contribute results. This matters because the "right" model depends entirely on what you're shipping on. A model that runs well on an H100 might be useless on a phone, and quantization choices affect both speed and accuracy in ways that need real-world testing.
Separately, they released DSpark draft models for their LFM2.5 family, which use speculative decoding to speed up inference. A small draft model proposes token blocks that the main model verifies in one pass. On an H100, the 8B model hits 3.18x speedup (428 to 1,362 tokens per second on MATH500), and on an M4 Max MacBook it reaches 2.87x (136 to 389 tokens per second). The speedup matters most for agentic workflows where the model has to reason before each tool call—on multi-tool scenarios, latency drops by nearly 50%. The draft models are around 300M parameters and share embeddings with their target models, so they add minimal memory overhead while keeping output identical to the baseline under greedy decoding.