CubeDAgger: Improved Robustness of Interactive Imitation Learning without Violation of Dynamic Stability
This work addresses dynamic stability issues in interactive imitation learning for robotics or autonomous systems, representing an incremental improvement over existing methods.
The paper tackles the problem of abrupt action changes and dynamic stability violations in interactive imitation learning by proposing CubeDAgger, which improves robustness through three enhancements to EnsembleDAgger, verified in simulations to maintain stability during interaction.
Interactive imitation learning makes an agent's control policy robust by stepwise supervisions from an expert. The recent algorithms mostly employ expert-agent switching systems to reduce the expert's burden by limitedly selecting the supervision timing. However, the precise selection is difficult and such a switching causes abrupt changes in actions, damaging the dynamic stability. This paper therefore proposes a novel method, so-called CubeDAgger, which improves robustness while reducing dynamic stability violations by making three improvements to a baseline method, EnsembleDAgger. The first improvement adds a regularization to explicitly activate the threshold for deciding the supervision timing. The second transforms the expert-agent switching system to an optimal consensus system of multiple action candidates. Third, autoregressive colored noise to the actions is introduced to make the stochastic exploration consistent over time. These improvements are verified by simulations, showing that the learned policies are sufficiently robust while maintaining dynamic stability during interaction.