Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learning
This addresses a more constrained and realistic setting for imitation learning, particularly useful for robotic tasks where environmental interactions are limited, though it appears incremental as it builds on existing methods with a novel framework.
The paper tackles the problem of strictly batch offline imitation learning, where the imitator cannot interact with the environment during training, by introducing a method based on the Markov balance equation and conditional density estimation. The result is consistently superior empirical performance compared to many state-of-the-art imitation learning algorithms, as demonstrated in experiments on Classic Control and MuJoCo environments.
Imitation learning (IL) is notably effective for robotic tasks where directly programming behaviors or defining optimal control costs is challenging. In this work, we address a scenario where the imitator relies solely on observed behavior and cannot make environmental interactions during learning. It does not have additional supplementary datasets beyond the expert's dataset nor any information about the transition dynamics. Unlike state-of-the-art (SOTA) IL methods, this approach tackles the limitations of conventional IL by operating in a more constrained and realistic setting. Our method uses the Markov balance equation and introduces a novel conditional density estimation-based imitation learning framework. It employs conditional normalizing flows for transition dynamics estimation and aims at satisfying a balance equation for the environment. Through a series of numerical experiments on Classic Control and MuJoCo environments, we demonstrate consistently superior empirical performance compared to many SOTA IL algorithms.