How your transcript knows where each word is spoken

Open original

ElevenLabs has released Cue, a forced alignment model that maps every character in an audio file to a precise timestamp. The article's case is that it beats open-source aligners on long files, noisy audio, language coverage, and speed.

  • On full recordings, Cue averages 179 ms of timing error, while the open-source tools tested ranged from 283 ms to over 6,000 ms.
  • It handles noisy English recordings with a 57 ms average error, versus 65 ms to 920 ms for the others, and processes an hour of audio in about half a second.
  • One model covers 29 benchmark languages in a single pass, so long files don't need chunking, which is where timing errors usually build up.

ElevenLabs has released Cue, a forced alignment model that maps every character in an audio file to a precise point on a timeline. Forced alignment is the job of matching a transcript to the moment each word is spoken, and it's what lets you click a word and have the player jump to that spot. Cue is the final stage of ElevenLabs' speech-to-text pipeline, sitting after the Scribe model transcribes audio, and it also drives subtitles, dubbing, and lyric timing. It's available on its own through a Forced Alignment API. The company's main claims are that Cue beats every aligner it tested on real-world recordings across all 29 languages in its benchmark, and that it can align an hour of audio in about half a second on a production GPU. The article's case against earlier tools rests on three problems the team heard from customers. First, length: many open-source aligners work well on clips around 30 seconds, so longer files get chunked, and errors build up at the seams. On full recordings Cue averaged 179 milliseconds of timing error, while the open-source tools ranged from 283 ms to over 6,000 ms. Second, noise: background sound, overlapping speakers, and music trip up many tools. On English recordings with background problems, Cue averaged 57 ms of error against 65 to 920 ms for the others. Third, language coverage: most open-source aligners need a separate model per language, so low-resource languages suffer or go unsupported. Cue is a single model covering more than 90 languages in total, and it was the only one besides MMS-FA to handle all 29 benchmark languages, where it was also generally faster at finding word boundaries. Cue is also smaller than ElevenLabs' previous aligner, which the team credits as a reason it runs so quickly. The article is candid about the limits of its own evaluation. Cue works at character level, but the benchmarks available mostly label words, so Kozlov says the scores probably understate its precision, and there's no golden benchmark for character-level alignment at sub-millisecond scale. Music is the other stress test: on a subset of JamendoLyrics songs in English, French, Spanish, and German, Cue placed 92.7% of words within 0.3 seconds of the label, against 82.0% for the next best tool, with an average start error of 0.82 seconds, roughly half the closest competitor's. The article also includes a short account of a Georgian-language regression a customer reported, which the team diagnosed, retrained, and shipped within 24 hours. Most of the numbers come from ElevenLabs' own benchmark, so they are worth checking against independent tests before you rely on them.

Ask about this article