Sounding Highlights: Dual-Pathway Audio Encoders for Audio-Visual Video Highlight Detection
This work addresses the underutilization of audio in highlight detection, offering improved accuracy for applications like video summarization, though it is incremental in enhancing existing multimodal methods.
The paper tackled the problem of audio-visual video highlight detection by proposing a dual-pathway audio encoder that captures both semantic content and dynamic spectro-temporal features, achieving new state-of-the-art performance on the Mr.HiSum benchmark.
Audio-visual video highlight detection aims to automatically identify the most salient moments in videos by leveraging both visual and auditory cues. However, existing models often underutilize the audio modality, focusing on high-level semantic features while failing to fully leverage the rich, dynamic characteristics of sound. To address this limitation, we propose a novel framework, Dual-Pathway Audio Encoders for Video Highlight Detection (DAViHD). The dual-pathway audio encoder is composed of a semantic pathway for content understanding and a dynamic pathway that captures spectro-temporal dynamics. The semantic pathway extracts high-level information by identifying the content within the audio, such as speech, music, or specific sound events. The dynamic pathway employs a frequency-adaptive mechanism as time evolves to jointly model these dynamics, enabling it to identify transient acoustic events via salient spectral bands and rapid energy changes. We integrate the novel audio encoder into a full audio-visual framework and achieve new state-of-the-art performance on the large-scale Mr.HiSum benchmark. Our results demonstrate that a sophisticated, dual-faceted audio representation is key to advancing the field of highlight detection.