What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning
This work addresses the challenge of understanding procedural activities in videos for applications in robotics and video analysis, though it is incremental by building on existing methods with new supervision signals.
The paper tackled the problem of learning video representations for procedural activities by explicitly modeling state changes and generating counterfactual scenarios, achieving significant improvements on tasks like temporal action segmentation and error detection.
Understanding a procedural activity requires modeling both how action steps transform the scene, and how evolving scene transformations can influence the sequence of action steps, even those that are accidental or erroneous. Existing work has studied procedure-aware video representations by modeling the temporal order of actions, but has not explicitly learned the state changes (scene transformations). In this work, we study procedure-aware video representation learning by incorporating state-change descriptions generated by Large Language Models (LLMs) as supervision signals for video encoders. Moreover, we generate state-change counterfactuals that simulate hypothesized failure outcomes, allowing models to learn by imagining unseen "What if" scenarios. This counterfactual reasoning facilitates the model's ability to understand the cause and effect of each step in an activity. We conduct extensive experiments on procedure-aware tasks, including temporal action segmentation, error detection, action phase classification, frame retrieval, multi-instance retrieval, and action recognition. Our results demonstrate the effectiveness of the proposed state-change descriptions and their counterfactuals, and achieve significant improvements on multiple tasks.