Imitation Learning from Observations under Transition Model Disparity
This addresses a domain-specific problem for robotics and reinforcement learning practitioners, offering an incremental improvement for handling dynamics mismatch in imitation learning.
The paper tackles imitation learning from observations when expert and learner environments have different transition dynamics, proposing an algorithm that trains an intermediary policy as a surrogate expert to match state transitions. Experiments on MuJoCo locomotion tasks show the method compares favorably to baselines, though no specific numerical gains are provided.
Learning to perform tasks by leveraging a dataset of expert observations, also known as imitation learning from observations (ILO), is an important paradigm for learning skills without access to the expert reward function or the expert actions. We consider ILO in the setting where the expert and the learner agents operate in different environments, with the source of the discrepancy being the transition dynamics model. Recent methods for scalable ILO utilize adversarial learning to match the state-transition distributions of the expert and the learner, an approach that becomes challenging when the dynamics are dissimilar. In this work, we propose an algorithm that trains an intermediary policy in the learner environment and uses it as a surrogate expert for the learner. The intermediary policy is learned such that the state transitions generated by it are close to the state transitions in the expert dataset. To derive a practical and scalable algorithm, we employ concepts from prior work on estimating the support of a probability distribution. Experiments using MuJoCo locomotion tasks highlight that our method compares favorably to the baselines for ILO with transition dynamics mismatch.