Learning to Localize and Align Fine-Grained Actions to Sparse Instructions
This addresses the challenge of bridging visual and language domains for time-aligned video descriptions, which is incremental as it builds on existing methods for action localization and alignment.
The paper tackles the problem of automatically aligning sparse textual instructions with first-person video demonstrations of activities, using egocentric cues to generate action proposals and matching them with object recognition and computational linguistics. They report promising results on the Extended GTEA Gaze+ and Bristol Egocentric Object Interactions datasets.
Automatic generation of textual video descriptions that are time-aligned with video content is a long-standing goal in computer vision. The task is challenging due to the difficulty of bridging the semantic gap between the visual and natural language domains. This paper addresses the task of automatically generating an alignment between a set of instructions and a first person video demonstrating an activity. The sparse descriptions and ambiguity of written instructions create significant alignment challenges. The key to our approach is the use of egocentric cues to generate a concise set of action proposals, which are then matched to recipe steps using object recognition and computational linguistic techniques. We obtain promising results on both the Extended GTEA Gaze+ dataset and the Bristol Egocentric Object Interactions Dataset.