Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding
This addresses a critical reliability issue in audio-language AI systems, though it is an incremental improvement using contrastive decoding.
The paper tackles object hallucination in Large Audio-Language Models by introducing Audio-Aware Decoding, an inference-time strategy that improves F1 scores by up to 0.428 on hallucination datasets and accuracy by up to 10.3% on general audio QA datasets.
Large Audio-Language Models (LALMs) can take audio and text as the inputs and answer questions about the audio. While prior LALMs have shown strong performance on standard benchmarks, there has been alarming evidence that LALMs can hallucinate what is presented in the audio. To mitigate the hallucination of LALMs, we introduce Audio-Aware Decoding (AAD), a lightweight inference-time strategy that uses contrastive decoding to compare the token prediction logits with and without the audio context. By contrastive decoding, AAD promotes the tokens whose probability increases when the audio is present. We conduct our experiment on object hallucination datasets with three LALMs and show that AAD improves the F1 score by 0.046 to 0.428. We also show that AAD can improve the accuracy on general audio QA datasets like Clotho-AQA by 5.4% to 10.3%. We conduct thorough ablation studies to understand the effectiveness of each component in AAD.