Model-based Offline Imitation Learning with Non-expert Data
This addresses the challenge of expensive optimal data collection in imitation learning by enabling use of cheaper suboptimal data, though it is incremental as it builds on existing model-based and offline learning frameworks.
The paper tackles the problem of imitation learning using both suboptimal and optimal datasets, showing that its worst-case suboptimality scales linearly with the time horizon relative to expert samples and always outperforms Behavioral Cloning in low-data regimes on simulated continuous control domains.
Although Behavioral Cloning (BC) in theory suffers compounding errors, its scalability and simplicity still makes it an attractive imitation learning algorithm. In contrast, imitation approaches with adversarial training typically does not share the same problem, but necessitates interactions with the environment. Meanwhile, most imitation learning methods only utilises optimal datasets, which could be significantly more expensive to obtain than its suboptimal counterpart. A question that arises is, can we utilise the suboptimal dataset in a principled manner, which otherwise would have been idle? We propose a scalable model-based offline imitation learning algorithmic framework that leverages datasets collected by both suboptimal and optimal policies, and show that its worst case suboptimality becomes linear in the time horizon with respect to the expert samples. We empirically validate our theoretical results and show that the proposed method \textit{always} outperforms BC in the low data regime on simulated continuous control domains