Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain
This addresses the challenge of domain transfer in inverse reinforcement learning for robotics and control applications, though it is incremental as it builds on existing GAN-based methods.
The paper tackles the problem of learning reward functions in simulators using real-world demonstrations by accounting for dynamics differences between domains, achieving effective performance on continuous control tasks with scalability to high-dimensional settings.
We propose an approach for inverse reinforcement learning from hetero-domain which learns a reward function in the simulator, drawing on the demonstrations from the real world. The intuition behind the method is that the reward function should not only be oriented to imitate the experts, but should encourage actions adjusted for the dynamics difference between the simulator and the real world. To achieve this, the widely used GAN-inspired IRL method is adopted, and its discriminator, recognizing policy-generating trajectories, is modified with the quantification of dynamics difference. The training process of the discriminator can yield the transferable reward function suitable for simulator dynamics, which can be guaranteed by derivation. Effectively, our method assigns higher rewards for demonstration trajectories which do not exploit discrepancies between the two domains. With extensive experiments on continuous control tasks, our method shows its effectiveness and demonstrates its scalability to high-dimensional tasks.