Dual-modality seq2seq network for audio-visual event localization
This work addresses audio-visual event localization for video analysis, presenting an incremental improvement with a novel hybrid method.
The authors tackled the problem of audio-visual event localization by proposing a dual-modality seq2seq network (AVSDN) that jointly processes audio and visual features, achieving favorable performance against recent deep learning approaches in both fully and weakly supervised settings.
Audio-visual event localization requires one to identify theevent which is both visible and audible in a video (eitherat a frame or video level). To address this task, we pro-pose a deep neural network named Audio-Visual sequence-to-sequence dual network (AVSDN). By jointly taking bothaudio and visual features at each time segment as inputs, ourproposed model learns global and local event information ina sequence to sequence manner, which can be realized in ei-ther fully supervised or weakly supervised settings. Empiricalresults confirm that our proposed method performs favorablyagainst recent deep learning approaches in both settings.