HCQA @ Ego4D EgoSchema Challenge 2024
This is an incremental improvement for researchers in egocentric video understanding, addressing the challenge of question answering in this domain.
The authors tackled the problem of answering multiple-choice questions about long-form egocentric videos by proposing HCQA, a hierarchical comprehension scheme that integrates captioning and reasoning models, achieving 75% accuracy on the EgoSchema test set.
In this report, we present our champion solution for Ego4D EgoSchema Challenge in CVPR 2024. To deeply integrate the powerful egocentric captioning model and question reasoning model, we propose a novel Hierarchical Comprehension scheme for egocentric video Question Answering, named HCQA. It consists of three stages: Fine-grained Caption Generation, Context-driven Summarization, and Inference-guided Answering. Given a long-form video, HCQA captures local detailed visual information and global summarised visual information via Fine-grained Caption Generation and Context-driven Summarization, respectively. Then in Inference-guided Answering, HCQA utilizes this hierarchical information to reason and answer given question. On the EgoSchema blind test set, HCQA achieves 75% accuracy in answering over 5,000 human curated multiple-choice questions. Our code will be released at https://github.com/Hyu-Zhang/HCQA.