DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering
This work improves egocentric video understanding for applications like assistive technologies or robotics by introducing a novel method to handle first-person perspective challenges, though it is incremental as it builds on pre-training and fine-tuning paradigms.
The paper tackles the problem of Egocentric Video Question Answering by addressing challenges like understanding multiple events and hand-object interactions in first-person videos, proposing a Dual-Modal Counterfactual Contrastive Construction (DMC^3) framework that achieves state-of-the-art performance with 52.51% on the normal split and 46.04% on the indirect split of EgoTaskQA, and 13.2% on QAEGO4D.
Egocentric Video Question Answering (Egocentric VideoQA) plays an important role in egocentric video understanding, which refers to answering questions based on first-person videos. Although existing methods have made progress through the paradigm of pre-training and fine-tuning, they ignore the unique challenges posed by the first-person perspective, such as understanding multiple events and recognizing hand-object interactions. To deal with these challenges, we propose a Dual-Modal Counterfactual Contrastive Construction (DMC$^3$) framework, which contains an egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization. Specifically, We first develop a counterfactual sample construction module to generate positive and negative samples for textual and visual modalities through event description paraphrasing and core interaction mining, respectively. Then, We feed these samples together with the original samples into the baseline. Finally, in the counterfactual sample-involved contrastive optimization module, we apply contrastive loss to minimize the distance between the original sample features and the positive sample features, while maximizing the distance from the negative samples. Experiments show that our method achieve 52.51\% and 46.04\% on the \textit{normal} and \textit{indirect} splits of EgoTaskQA, and 13.2\% on QAEGO4D, both reaching the state-of-the-art performance.