Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement
This work addresses the problem of scalable imitation learning for AI systems by enabling learning from action-free demonstrations, though it appears incremental as it adapts existing RL-based methods to more complex data distributions.
The paper tackles the challenge of Imitation Learning from Observation (IfO) by addressing nuanced data distributions beyond idealized bimodal ones, introducing a method that uses a value function to transfer information between expert and non-expert data, achieving results that move towards iterative self-improvement for scalable behavior learning.
Imitation Learning from Observation (IfO) offers a powerful way to learn behaviors at large-scale: Unlike behavior cloning or offline reinforcement learning, IfO can leverage action-free demonstrations and thus circumvents the need for costly action-labeled demonstrations or reward functions. However, current IfO research focuses on idealized scenarios with mostly bimodal-quality data distributions, restricting the meaningfulness of the results. In contrast, this paper investigates more nuanced distributions and introduces a method to learn from such data, moving closer to a paradigm in which imitation learning can be performed iteratively via self-improvement. Our method adapts RL-based imitation learning to action-free demonstrations, using a value function to transfer information between expert and non-expert data. Through comprehensive evaluation, we delineate the relation between different data distributions and the applicability of algorithms and highlight the limitations of established methods. Our findings provide valuable insights for developing more robust and practical IfO techniques on a path to scalable behaviour learning.