Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations
This work addresses the challenge of understanding procedural activities in videos for applications like automated instruction analysis, with incremental but strong gains in specific tasks.
The paper tackles the problem of learning video representations for procedural activities from instructional videos and narrations without human annotations, achieving state-of-the-art improvements in step classification (+2.8% on COIN, +3.3% on EPIC-Kitchens) and step forecasting (+7.4% on COIN).
The abundance of instructional videos and their narrations over the Internet offers an exciting avenue for understanding procedural activities. In this work, we propose to learn video representation that encodes both action steps and their temporal ordering, based on a large-scale dataset of web instructional videos and their narrations, without using human annotations. Our method jointly learns a video representation to encode individual step concepts, and a deep probabilistic model to capture both temporal dependencies and immense individual variations in the step ordering. We empirically demonstrate that learning temporal ordering not only enables new capabilities for procedure reasoning, but also reinforces the recognition of individual steps. Our model significantly advances the state-of-the-art results on step classification (+2.8% / +3.3% on COIN / EPIC-Kitchens) and step forecasting (+7.4% on COIN). Moreover, our model attains promising results in zero-shot inference for step classification and forecasting, as well as in predicting diverse and plausible steps for incomplete procedures. Our code is available at https://github.com/facebookresearch/ProcedureVRL.