Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
Addresses the practical problem of ASR hallucinations for users of Whisper models in real-world audio processing.
Whisper ASR models hallucinate on non-speech audio. Using SAE-based steering, hallucination rate dropped from 72.63% to 14.11% (Whisper small) and 86.88% to 27.33% (Whisper large-v3) with minimal WER degradation on speech.
Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input. We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations. We extract audio encoder activations and evaluate two representation spaces: raw Whisper activations and Sparse AutoEncoder (SAE) latents. We show that both spaces encode linearly separable hallucination-related information, with discriminative power concentrated in a sparse feature subset and increasing toward deeper encoder layers. We propose two steering strategies: activation-space steering and SAE latent-space steering. SAE-based steering reduces hallucination rate from 72.63% to 14.11% for Whisper small and from 86.88% to 27.33% for Whisper large-v3 on the full non-speech test set, with small WER degradation on speech data, approaching the performance of fine-tuning-based methods.