Characterizing Video Question Answering with Sparsified Inputs
This work addresses data efficiency for video-and-language tasks, but it is incremental as it extends prior single-frame selection to multiple inputs and modalities.
The paper tackles the problem of using sparsified inputs in Video Question Answering to improve data efficiency, showing that selecting only 10% of video lengths (2-4 frames) results in a performance loss of only 5.2%-5.8% on public benchmarks.
In Video Question Answering, videos are often processed as a full-length sequence of frames to ensure minimal loss of information. Recent works have demonstrated evidence that sparse video inputs are sufficient to maintain high performance. However, they usually discuss the case of single frame selection. In our work, we extend the setting to multiple number of inputs and other modalities. We characterize the task with different input sparsity and provide a tool for doing that. Specifically, we use a Gumbel-based learnable selection module to adaptively select the best inputs for the final task. In this way, we experiment over public VideoQA benchmarks and provide analysis on how sparsified inputs affect the performance. From our experiments, we have observed only 5.2%-5.8% loss of performance with only 10% of video lengths, which corresponds to 2-4 frames selected from each video. Meanwhile, we also observed the complimentary behaviour between visual and textual inputs, even under highly sparsified settings, suggesting the potential of improving data efficiency for video-and-language tasks.