Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning
For researchers in multimodal video understanding, this work provides a novel framework and benchmark to improve the alignment of auditory and visual information in generated captions.
The paper addresses the challenge of temporal and cross-modal alignment in audiovisual video captioning, proposing TCA-Captioner which achieves new state-of-the-art performance on multiple benchmarks, with significant improvements in audiovisual binding and temporal coherence.
While Multimodal Large Language Models (MLLMs) have advanced video understanding, achieving precise temporal and cross-modal alignment in audiovisual video captioning remains a formidable challenge. Most existing approaches suffer from modality detachment and temporal incoherence, failing to accurately bind auditory events to visual entities or capture complex causal dynamics. To address these deficiencies, we propose TCA-Captioner, a framework specifically engineered to enhance Temporal and Cross-Modal Alignment for audiovisual video captioning. We first introduce the Observer-Checker-Corrector (OCC) framework, an iterative refinement strategy that generates high-fidelity, meticulously grounded training data. Leveraging a curated high-density human interaction dataset, TCA-Captioner is optimized to model sophisticated audiovisual interactions. Furthermore, we present TCA-Bench, a diagnostic benchmark utilizing a Decoupled Evaluation Protocol to isolate and quantify model proficiency in audiovisual binding and temporal relational reasoning. Extensive experiments demonstrate that TCA-Captioner sets a new standard for temporally-coherent and synchronized audiovisual narratives.