Click any tag below to further narrow down your results
+ classification
(2)
+ inference speed
(2)
+ eleven v4
(1)
+ cost benchmark
(1)
+ content moderation
(1)
+ policylm 1.7b
(1)
+ multilingual
(1)
+ real time speech
(1)
+ voice agents
(1)
+ localization
(1)
+ emotional expression
(1)
+ text to speech
(1)
+ jev
(1)
+ vendor lock in
(1)
+ infrastructure
(1)
Links
Airbnb benchmarked a fine-tuned 2B-parameter classifier (E2B) against frontier and open-weight APIs on a real policy-screening task. The small model matched the best proprietary models on F1 while returning in about 35ms median, at roughly a hundredth of the cost at scale.
- Fine-tuning Gemma4-E2B on hundreds of thousands of in-domain examples raised F1 from 0.563 to 0.902, on par with frontier models, with median latency around 35ms versus about 900ms for frontier APIs.
- A 2,700-word detailed prompt only gained one to three F1 points for proprietary models and did not close the gap for models fast enough to deploy, so prompt tuning alone was not the answer.
- Self-hosting one GPU beats frontier APIs on cost above roughly 15 to 30 requests per minute; at Airbnb's volume, frontier APIs cost about two orders of magnitude more.
Martin Alderson walks through what he's learned about speeding up agentic AI workflows, drawing on his web performance background. The core argument is that model inference is often not the biggest bottleneck; tool calls, shell startup, polling loops, and hardware choices frequently matter more.
- Inference has two parts to watch: time to first token (dominated by prefill and provider queueing) and tokens per second. Which one matters depends on whether your agent sends short back-and-forth turns or long outputs, and thinking budget can dwarf both.
- Halving inference time only cuts about 20% off a task if provisioning, tool calls, and suspend/resume stay the same. Tool call traces often reveal slow things like xargs spawning hundreds of thousands of processes or a slow .bashrc (nvm is a common culprit).
- Polling loops with sleep are a red flag; the agent is waiting for something that should push a notification instead. Faster underlying hardware (CPU type, local NVMe versus networked disks) and careful RAM management during suspend/resume also make a measurable difference.
Musubi Labs released PolicyLM-1.7B, a small open-source language model designed to make moderation decisions on live messages in under 100ms by reading your platform's actual policies instead of relying on fixed taxonomies. It runs on a single GPU or laptop and lets teams update policies instantly without retraining.
- Processes messages in under 100ms (median 35ms on short chat) while scoring against custom policies, making it viable for live chat where fixed classifiers are blind to policy changes and large LLMs are too slow.
- Reads your policy with every message, so policy edits take effect immediately without retraining or annotation work—useful for teams with frequent rule changes or policies that don't fit standard taxonomies.
- Open weights (Apache 2.0) and already proven in production on a platform handling over a million messages daily; includes two threshold presets (precision-focused for rare violations, balanced for higher detection).
Microsoft released three new models for building voice agents: a streaming transcription model that produces text in 100ms, and two text-to-speech models optimized for speed and cost. Together they're designed to handle the full loop of hearing, understanding, deciding, and speaking in real-time conversations.
- MAI-Transcribe-2-Streaming ranks #1 for accuracy on Artificial Analysis and delivers partial transcripts in just over 100ms, letting voice agents start acting mid-sentence rather than waiting for speakers to finish.
- MAI-Voice-2.1-Flash generates 45 seconds of audio with 150ms end-to-end latency and costs 60% less than comparable models at $15 per 1M characters, making it practical for high-volume applications.
- Both voice models support 23 languages with a single voice maintaining native accents across all of them, plus voice cloning from a few seconds of audio with built-in consent guardrails against misuse.
ElevenLabs released Eleven v4, a text-to-speech model that generates emotionally expressive speech by interpreting tone, pacing, and context. It ranks #1 in independent benchmarks and was preferred by ~75% of listeners in blind tests against competitors.
- Eleven v4 can generate speech with distinct emotions (dramatic, tender, urgent, comedic) while maintaining speaker identity, and users can control delivery through natural language prompts and inline tags like [laughs] or [said angrily in French accent]
- Eleven v4 Turbo achieves ~150ms latency for conversational agents, solving the traditional speed-vs-quality tradeoff that forced developers to choose between fast but robotic or slow but expressive voices
- The model supports 90+ languages and can clone voices that speak fluently in any language while adopting native accents and retaining the original voice's identity, improving dubbing and localization work
S3 became the default storage layer for cloud systems because of disk constraints 20 years ago, but SSDs and modern networks have made those constraints irrelevant. Cloud providers won't build a faster replacement because they profit from the status quo, so someone else needs to.
- S3's architecture is fundamentally a workaround for slow disk performance (100ms latency, <100MB/s per request), forcing every system built on it to add caching layers, batching logic, and separate metadata stores.
- SSDs are now only 3× more expensive than disks while offering microsecond latency, millions of IOPS, and small random access—eliminating the need for most S3-based workarounds.
- Amazon's S3 Express One Zone proves the company won't cannibalize its disk-based S3 revenue stream with a true SSD replacement, leaving the door open for competitors to build it.
Jev is a new AI model that only outputs structured data instead of human language, making it dramatically faster (70-500ms vs seconds) and enabling real-time applications like playing Doom. The author argues this speed advantage could become a new computational primitive for AI, though he suspects competitors can replicate it using simpler inference tricks on existing models.
- Jev generates all structured output in a single forward pass instead of token-by-token, achieving 70-500ms response times compared to seconds for standard LLMs, fast enough to play real-time video games.
- The speed advantage likely doesn't require novel model architecture—you can achieve similar results by prefilling responses and generating only one constrained token with existing LLMs, suggesting Jev lacks a substantial technical moat.
- Structured output could unlock entirely new use cases beyond chatbots by injecting "100ms worth of dirt-cheap intelligence" at decision points throughout applications, though Jev won't match frontier LLMs in raw capability.