When does predictive inverse dynamics outperform behavior cloning?
This work provides a theoretical explanation for when PIDM outperforms BC, offering insights for improving imitation learning in robotics and gaming, though it is incremental as it builds on existing PIDM architectures.
The paper tackles the problem of offline imitation learning with limited expert demonstrations by analyzing predictive inverse dynamics models (PIDM) versus behavior cloning (BC), showing that PIDM reduces variance at the cost of bias and achieves higher sample efficiency, requiring up to five times fewer demonstrations in 2D tasks and over 66% fewer samples in a 3D environment.
Behavior cloning (BC) is a practical offline imitation learning method, but it often fails when expert demonstrations are limited. Recent works have introduced a class of architectures named predictive inverse dynamics models (PIDM) that combine a future state predictor with an inverse dynamics model (IDM). While PIDM often outperforms BC, the reasons behind its benefits remain unclear. In this paper, we provide a theoretical explanation: PIDM introduces a bias-variance tradeoff. While predicting the future state introduces bias, conditioning the IDM on the prediction can significantly reduce variance. We establish conditions on the state predictor bias for PIDM to achieve lower prediction error and higher sample efficiency than BC, with the gap widening when additional data sources are available. We validate the theoretical insights empirically in 2D navigation tasks, where BC requires up to five times (three times on average) more demonstrations than PIDM to reach comparable performance; and in a complex 3D environment in a modern video game with high-dimensional visual inputs and stochastic transitions, where BC requires over 66\% more samples than PIDM.