MILE: Model-based Intervention Learning
This work addresses the challenge of reducing expert effort in imitation learning for robotics, though it is incremental by building on existing interactive methods.
The paper tackles the problem of imitation learning requiring complete expert trajectories and suffering from compounding errors by proposing a model that learns from both intervention and non-intervention timesteps, achieving effective policy learning with only a handful of expert interventions across simulation environments, a robotic manipulation task, and a human study.
Imitation learning techniques have been shown to be highly effective in real-world control scenarios, such as robotics. However, these approaches not only suffer from compounding error issues but also require human experts to provide complete trajectories. Although there exist interactive methods where an expert oversees the robot and intervenes if needed, these extensions usually only utilize the data collected during intervention periods and ignore the feedback signal hidden in non-intervention timesteps. In this work, we create a model to formulate how the interventions occur in such cases, and show that it is possible to learn a policy with just a handful of expert interventions. Our key insight is that it is possible to get crucial information about the quality of the current state and the optimality of the chosen action from expert feedback, regardless of the presence or the absence of intervention. We evaluate our method on various discrete and continuous simulation environments, a real-world robotic manipulation task, as well as a human subject study. Videos and the code can be found at https://liralab.usc.edu/mile .