Can Hallucination Correction Improve Video-Language Alignment?
This addresses the issue of misalignment between video and text for applications in video understanding and retrieval, though it is incremental as it builds on prior hallucination mitigation work.
The paper tackles the problem of hallucinated content in large vision-language models by using hallucination correction as a training objective to improve video-language alignment, resulting in consistent gains in video-caption binding and text-to-video retrieval tasks.
Large Vision-Language Models often generate hallucinated content that is not grounded in its visual inputs. While prior work focuses on mitigating hallucinations, we instead explore leveraging hallucination correction as a training objective to improve video-language alignment. We introduce HACA, a self-training framework learning to correct hallucinations in descriptions that do not align with the video content. By identifying and correcting inconsistencies, HACA enhances the model's ability to align video and textual representations for spatio-temporal reasoning. Our experimental results show consistent gains in video-caption binding and text-to-video retrieval tasks, demonstrating that hallucination correction-inspired tasks serve as an effective strategy for improving vision and language alignment.