Efficient End-to-End Video Question Answering with Pyramidal Multimodal Transformer
This addresses video question answering for AI systems, presenting an incremental improvement through a novel architecture that balances performance and efficiency.
The paper tackles video question answering by proposing a pyramidal multimodal transformer that processes video-language interactions across spatio-temporal scales, achieving better or on-par performance with state-of-the-art methods on five benchmarks while maintaining high computational efficiency.
This paper presents a new method for end-to-end Video Question Answering (VideoQA), aside from the current popularity of using large-scale pre-training with huge feature extractors. We achieve this with a pyramidal multimodal transformer (PMT) model, which simply incorporates a learnable word embedding layer, a few convolutional and transformer layers. We use the anisotropic pyramid to fulfill video-language interactions across different spatio-temporal scales. In addition to the canonical pyramid, which includes both bottom-up and top-down pathways with lateral connections, novel strategies are proposed to decompose the visual feature stream into spatial and temporal sub-streams at different scales and implement their interactions with the linguistic semantics while preserving the integrity of local and global semantics. We demonstrate better or on-par performances with high computational efficiency against state-of-the-art methods on five VideoQA benchmarks. Our ablation study shows the scalability of our model that achieves competitive results for text-to-video retrieval by leveraging feature extractors with reusable pre-trained weights, and also the effectiveness of the pyramid.