Is a Video worth $n\times n$ Images? A Highly Efficient Approach to Transformer-based Video Question Answering
This work addresses computational efficiency for researchers with limited budgets in VideoQA, though it is incremental as it builds on existing vision-language models.
The paper tackles the high computational cost of Transformer-based Video Question Answering by concatenating video frames into a single image, reducing image encoder usage from n² to 1 while preserving temporal structure. It achieves state-of-the-art performance on MSRVTT and TrafficQA with nearly 4× faster speed and 30% memory use.
Conventional Transformer-based Video Question Answering (VideoQA) approaches generally encode frames independently through one or more image encoders followed by interaction between frames and question. However, such schema would incur significant memory use and inevitably slow down the training and inference speed. In this work, we present a highly efficient approach for VideoQA based on existing vision-language pre-trained models where we concatenate video frames to a $n\times n$ matrix and then convert it to one image. By doing so, we reduce the use of the image encoder from $n^{2}$ to $1$ while maintaining the temporal structure of the original video. Experimental results on MSRVTT and TrafficQA show that our proposed approach achieves state-of-the-art performance with nearly $4\times$ faster speed and only 30% memory use. We show that by integrating our approach into VideoQA systems we can achieve comparable, even superior, performance with a significant speed up for training and inference. We believe the proposed approach can facilitate VideoQA-related research by reducing the computational requirements for those who have limited access to budgets and resources. Our code will be made publicly available for research use.