Accent related errors in clinical speech transcription and a LLM-based remedy - npj Digital Medicine

Open original

This pilot study tested Whisper and WhisperX on standardized medical recordings from native and non-native English speakers and found much higher error rates for non-native speakers. Adding a GPT-4o correction step cut word error rates by up to 6.9% and nearly eliminated accent-related disparities. The chained ASR–LLM pipeline boosts transcription accuracy across diverse clinical speakers.

  • Whisper and WhisperX both show significantly higher word error rates for non-native English speakers on clinical scripts, with Whisper's gap being larger.
  • Complex terminology in cardiology/pulmonology scripts hit accuracy hardest, dropping NER precision and F1 for non-native speech.
  • Piping WhisperX output through GPT-4o for correction cut WER by up to 6.9% and nearly closed the native/non-native accuracy gap.
  • With ~1 in 5 US healthcare workers foreign-born, uncorrected ASR bias risks misrecorded diagnoses or medication names in AI scribe tools.

Clinical speech recognition models like Whisper and its enhanced variant WhisperX struggle with non-native English accents, leading to higher transcription errors in medical settings. In a pilot study, researchers recorded four standardized clinical scripts—ranging from a gastroenterologist’s colonoscopy explanation to inter-physician histories from cardiology, gastroenterology, and pulmonology—and ran them through both ASR systems. Word Error Rates (WER) and Levenshtein distances were markedly worse for non-native speakers; Whisper’s gap was larger, but WhisperX still showed significant disparity. A specialized biomedical NER model revealed that complex terms in the pulmonology and cardiology texts suffered most, dropping precision and F1 scores compared to simpler scenarios. Beyond raw error counts, semantic measures told a similar story. WhisperX maintained strong BERTScores across all texts—indicating overall meaning was often preserved despite mistakes—but non-native segments still saw degraded scores. To tackle this, the team piped WhisperX outputs into GPT-4o for post-processing. That chained pipeline slashed accent-related errors, restoring parity between native and non-native transcripts and improving NER recall by notable margins. The paper reports that GPT-4o corrections brought non-native WER back in line with native speaker performance, though exact numbers vary by script complexity. These findings matter because nearly one in five U.S. healthcare workers is foreign-born and increasingly relies on AI scribes for real-time documentation. Left unchecked, ASR bias could introduce clinical risk through misrecorded diagnoses or medication names. By combining a top-tier ASR engine with a powerful LLM for correction, the study offers a concrete approach to leveling the playing field across diverse accents—a step toward safer, more equitable AI tools in medicine.

Ask about this article