From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection
For ASR practitioners, this work provides a comparative analysis of hallucination detection methods, showing that internal model states can outperform text-based approaches without requiring reference transcripts.
The paper studies hallucination detection in Whisper ASR across text-based, LLM-based, and internal decoder state probing methods, finding that probing decoder representations without reference transcripts yields the strongest performance, and a late-fusion meta-classifier achieves the best overall detection.
Hallucinations of ASR models - fluent transcriptions with no basis in audio - degrade system performance and pose risks in downstream applications. Robust detection of such errors remains a challenge. This paper studies Whisper large v3 hallucination detection on real-speech human-annotated data across three paradigms: text-based, LLM-based, and internal decoder state probing. Text classifiers utilizing metrics for text evaluation achieve high recall but degrade without reference transcripts. LLM-based detection improves precision with domain-specific prompt conditioning, yet remains less competitive than the lightweight text-based methods. Probing Whisper's decoder representations, without a ground-truth reference, yields the strongest performance, revealing that hallucination traits are encoded across intermediate decoding layers. A late-fusion meta-classifier combining text and internal-state outputs achieves the best overall detection performance.