P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision
This addresses the challenge of reducing annotation costs for training AI agents in procedural tasks, though it is incremental as it builds on existing planning methods.
The paper tackles the problem of procedure planning in instructional videos by proposing a weakly supervised approach that uses natural language instructions instead of expensive temporal annotations, and shows that it outperforms fully supervised state-of-the-art models on three datasets.
In this paper, we study the problem of procedure planning in instructional videos. Here, an agent must produce a plausible sequence of actions that can transform the environment from a given start to a desired goal state. When learning procedure planning from instructional videos, most recent work leverages intermediate visual observations as supervision, which requires expensive annotation efforts to localize precisely all the instructional steps in training videos. In contrast, we remove the need for expensive temporal video annotations and propose a weakly supervised approach by learning from natural language instructions. Our model is based on a transformer equipped with a memory module, which maps the start and goal observations to a sequence of plausible actions. Furthermore, we augment our model with a probabilistic generative module to capture the uncertainty inherent to procedure planning, an aspect largely overlooked by previous work. We evaluate our model on three datasets and show our weaklysupervised approach outperforms previous fully supervised state-of-the-art models on multiple metrics.