What to Do Next? Memorizing skills from Egocentric Instructional Video
This work addresses the challenge of learning skills from demonstrations for applications in robotics or AI assistants, but it appears incremental as it builds on existing memory and transformer techniques.
The paper tackles the problem of planning high-level goal-oriented actions from egocentric instructional videos by introducing an interactive action planning task and a method combining topological affordance memory with transformers, resulting in improved performance and robustness to action deviations in a simulation environment.
Learning to perform activities through demonstration requires extracting meaningful information about the environment from observations. In this research, we investigate the challenge of planning high-level goal-oriented actions in a simulation setting from an egocentric perspective. We present a novel task, interactive action planning, and propose an approach that combines topological affordance memory with transformer architecture. The process of memorizing the environment's structure through extracting affordances facilitates selecting appropriate actions based on the context. Moreover, the memory model allows us to detect action deviations while accomplishing specific objectives. To assess the method's versatility, we evaluate it in a realistic interactive simulation environment. Our experimental results demonstrate that the proposed approach learns meaningful representations, resulting in improved performance and robust when action deviations occur.