PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Reward
This addresses a critical issue in imitation learning for robotics or AI systems where reward misalignment leads to poor performance, though it appears incremental as it builds on existing IRL-based IL methods.
The paper tackles the problem of reward misalignment in inverse reinforcement learning-based imitation learning, which can cause task failures, by introducing PAGAR, a semi-supervised reward design paradigm that trains policies under mixed reward functions. Experimental results show that the algorithm outperforms standard baselines in complex tasks and challenging transfer settings.
Many imitation learning (IL) algorithms employ inverse reinforcement learning (IRL) to infer the intrinsic reward function that an expert is implicitly optimizing for based on their demonstrated behaviors. However, in practice, IRL-based IL can fail to accomplish the underlying task due to a misalignment between the inferred reward and the objective of the task. In this paper, we address the susceptibility of IL to such misalignment by introducing a semi-supervised reward design paradigm called Protagonist Antagonist Guided Adversarial Reward (PAGAR). PAGAR-based IL trains a policy to perform well under mixed reward functions instead of a single reward function as in IRL-based IL. We identify the theoretical conditions under which PAGAR-based IL can avoid the task failures caused by reward misalignment. We also present a practical on-and-off policy approach to implementing PAGAR-based IL. Experimental results show that our algorithm outperforms standard IL baselines in complex tasks and challenging transfer settings.