Top-down Activity Representation Learning for Video Question Answering
This addresses the challenge of modeling non-continuous contextual events in videos for improved video question answering, representing an incremental advance.
The paper tackled the problem of capturing hierarchical human activities in videos for video question answering by converting long-term video sequences into spatial images and finetuning LLaVA, achieving a 78.4% accuracy on NExTQA, which exceeds the state-of-the-art by 2.8 points.
Capturing complex hierarchical human activities, from atomic actions (e.g., picking up one present, moving to the sofa, unwrapping the present) to contextual events (e.g., celebrating Christmas) is crucial for achieving high-performance video question answering (VideoQA). Recent works have expanded multimodal models (e.g., CLIP, LLaVA) to process continuous video sequences, enhancing the model's temporal reasoning capabilities. However, these approaches often fail to capture contextual events that can be decomposed into multiple atomic actions non-continuously distributed over relatively long-term sequences. In this paper, to leverage the spatial visual context representation capability of the CLIP model for obtaining non-continuous visual representations in terms of contextual events in videos, we convert long-term video sequences into a spatial image domain and finetune the multimodal model LLaVA for the VideoQA task. Our approach achieves competitive performance on the STAR task, in particular, with a 78.4% accuracy score, exceeding the current state-of-the-art score by 2.8 points on the NExTQA task.