Towards Good Practices for Multi-modal Fusion in Large-scale Video Classification
This work addresses the challenge of effective multi-modal fusion for video classification, which is incremental as it builds on existing bilinear pooling techniques.
The paper tackled the problem of improving large-scale video classification by proposing multi-modal factorized bilinear pooling (MFB) to fuse visual and audio representations, and it demonstrated that MFB significantly outperforms simple fusion methods on the Youtube-8M v2 dataset.
Leveraging both visual frames and audio has been experimentally proven effective to improve large-scale video classification. Previous research on video classification mainly focuses on the analysis of visual content among extracted video frames and their temporal feature aggregation. In contrast, multimodal data fusion is achieved by simple operators like average and concatenation. Inspired by the success of bilinear pooling in the visual and language fusion, we introduce multi-modal factorized bilinear pooling (MFB) to fuse visual and audio representations. We combine MFB with different video-level features and explore its effectiveness in video classification. Experimental results on the challenging Youtube-8M v2 dataset demonstrate that MFB significantly outperforms simple fusion methods in large-scale video classification.