Summary of StepAudio 3 Gen Technical Report StepAudio 3 Gen is a unified audio generation model that handles text-to-speech, voice design, vocals, sound effects, music, and mixed audio types all in one system. Instead of following the recent trend toward diffusion-based approaches, the researchers went with discrete autoregressive generation—meaning it predicts audio tokens one step at a time. The model uses residual vector quantization (RVQ) with 16 codebook layers at 12.5 Hz, where each layer captures both semantic and acoustic information. The generation process works in two stages: first predicting the primary codebook autoregressively along the time axis, then using a lightweight causal Transformer to fill in the remaining 15 codebooks along the codebook dimension. The technical approach hinges on three design choices that set it apart. They use interference-aware progressive pretraining to add audio capabilities without erasing the language understanding the base model already has—basically, you don't want the audio training to break the text comprehension. An RVQ Adaptor helps the system effectively work with multiple codebook layers at once. The discrete autoregressive modeling across a shared representation keeps everything unified rather than splitting into separate domain-specific paths. This matters because it means one model learns to handle radically different audio types without fragmenting into specialized branches. The results show state-of-the-art performance on both TTS and voice design tasks, while maintaining strong output across speech, vocals, effects, and music. They combined progressive pretraining, multi-task instruction training, and supervised fine-tuning to get there. Audio samples are available on their demo page, so you can actually hear what the model produces rather than just reading claims about it.