Bringing Your Muse to Life

Open original

Meta released Muse Realtime Avatar, a system that generates expressive video avatars synchronized with AI speech in real time. It achieves 870ms latency while outperforming existing avatar products from Runway and HeyGen in user preference tests.

  • The system reduces computational cost by 60x through distillation, condensing a teacher model's 120 diffusion steps into a student model's 2 unguided steps while preserving quality.
  • Muse generates 448x768 video at 25 fps with ~870ms end-to-end latency, handling 12 concurrent sessions per GPU through optimized inference, KV cache reuse, and four-bit quantization.
  • Head-to-head comparisons with leading commercial avatar systems showed raters preferred Muse Realtime Avatar across visual quality, synchronization, character consistency, and mannerisms.

Meta just released Muse Realtime Avatar, a system that animates static images or illustrations in real time during conversations. Feed it a photo, a drawing, or even an object, and it'll generate synchronized video of that thing speaking—with natural expressions, gestures, and head movements that stay consistent throughout the chat. The tech works by tapping into the same token stream that powers Muse Realtime Voice, so the avatar's lip sync and expressions match the speech automatically. This unified approach means voice, mouth movement, and body language all stay locked together without extra coordination work. The engineering challenge here is brutal: generate smooth, coherent video fast enough for real-time chat without errors piling up as the conversation goes on. Meta built a Diffusion Transformer that works in short chunks, where each new piece of video feeds context into the next one, keeping the computation bounded so it can run indefinitely. They started with a teacher model using 120 evaluations per chunk, then distilled it down to a student model that does the same job in just two evaluations—a 60x speedup—by training the student on its own mistakes so it learns to resist drift. The results beat Runway Characters and HeyGen LiveAvatar in head-to-head comparisons on visual quality, sync, and consistency. At scale, the system streams 448x768 video at 25 fps with about 870 milliseconds of latency from when you finish talking to when the avatar's response arrives. On a single NVIDIA GB200 chip, they squeezed in 12 concurrent sessions by redesigning Meta's real-time inference stack end-to-end—persistent KV caches, cache-aware routing, four-bit quantization, fused kernels. Each generation step takes 20 milliseconds to produce eight frames of playback. On the safety side, Meta embeds an invisible watermark in all generated video using Meta Video Seal to make deepfakes traceable, and the system is restricted to users 18 and older.

Ask about this article