Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding
This work provides a significant improvement in accuracy and speed for long-form speech recognition, particularly benefiting users of deployed Whisper systems by offering a drop-in inference-time solution.
This paper addresses hallucinations, repetition loops, and content omissions in long-form speech recognition by large encoder-decoder models like Whisper. The proposed training-free contrastive decoding framework, Whisper-CD, reduces Word Error Rate (WER) by up to 24.3 percentage points on CORAAL and achieves 48% faster token generation throughput compared to beam search across five English long-form benchmarks.
Long-form speech recognition with large encoder-decoder models such as Whisper often exhibit hallucinations, repetition loops, and content omissions. These errors can accumulate and be further amplified when the previous segment's transcription is used as decoding context. We propose Whisper-CD, a training-free contrastive decoding framework that contrasts clean-audio logits against negative logits computed from three acoustically motivated perturbations: Gaussian noise injection, silence signal, and audio temporal shift. We aggregate these negatives via the log-sum-exp operator, building a unified multi-negative objective for token-by-token decoding. Across five English long-form benchmarks, Whisper-CD reduces WER by up to 24.3pp on CORAAL and shows 48% faster token generation throughput than beam search. Because Whisper-CD operates purely at inference time, it can be applied as a drop-in replacement to already-deployed Whisper systems without retraining.