MPN: Multimodal Parallel Network for Audio-Visual Event Localization
This addresses a widespread audio-visual scene analysis problem for unconstrained video applications, representing an incremental improvement with novel attention modules.
The paper tackles audio-visual event localization in unconstrained videos by proposing a Multimodal Parallel Network (MPN) that processes global semantics and local information in parallel, achieving state-of-the-art performance on the AVE dataset in both fully and weakly supervised settings.
Audio-visual event localization aims to localize an event that is both audible and visible in the wild, which is a widespread audio-visual scene analysis task for unconstrained videos. To address this task, we propose a Multimodal Parallel Network (MPN), which can perceive global semantics and unmixed local information parallelly. Specifically, our MPN framework consists of a classification subnetwork to predict event categories and a localization subnetwork to predict event boundaries. The classification subnetwork is constructed by the Multimodal Co-attention Module (MCM) and obtains global contexts. The localization subnetwork consists of Multimodal Bottleneck Attention Module (MBAM), which is designed to extract fine-grained segment-level contents. Extensive experiments demonstrate that our framework achieves the state-of-the-art performance both in fully supervised and weakly supervised settings on the Audio-Visual Event (AVE) dataset.