Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
This work addresses the problem of enabling AI to learn decision-making from human videos, which is incremental as it builds on prior methods but introduces novel modeling to improve planning accuracy.
The paper tackles the problem of learning to plan goal-directed actions from instructional videos by addressing limitations of previous world models that fail to distinguish tasks and degrade goal information. It proposes a new formulation using Bayesian inference and model-based imitation learning, achieving state-of-the-art performance in reaching indicated goals on real-world videos.
Learning new skills by observing humans' behaviors is an essential capability of AI. In this work, we leverage instructional videos to study humans' decision-making processes, focusing on learning a model to plan goal-directed actions in real-life videos. In contrast to conventional action recognition, goal-directed actions are based on expectations of their outcomes requiring causal knowledge of potential consequences of actions. Thus, integrating the environment structure with goals is critical for solving this task. Previous works learn a single world model will fail to distinguish various tasks, resulting in an ambiguous latent space; planning through it will gradually neglect the desired outcomes since the global information of the future goal degrades quickly as the procedure evolves. We address these limitations with a new formulation of procedure planning and propose novel algorithms to model human behaviors through Bayesian Inference and model-based Imitation Learning. Experiments conducted on real-world instructional videos show that our method can achieve state-of-the-art performance in reaching the indicated goals. Furthermore, the learned contextual information presents interesting features for planning in a latent space.