From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
For researchers in egocentric action recognition, this work provides a fine-tuning-free alternative that leverages VLMs' symbolic reasoning strengths, though the gains are incremental over existing methods.
The authors propose converting egocentric videos into Temporal Action Graphs to decouple visual perception from symbolic reasoning, enabling VLMs to perform action recognition via in-context learning. On EGTEA and Epic-Kitchens-100, few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame and graph-based inference, and zero-shot graph reasoning remains competitive with pixel-based inference across 11 VLMs.
Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame and graph-based inference alike. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.