Towards Resolving Unidentifiability in Inverse Reinforcement Learning
This addresses a fundamental limitation in IRL for robotics and AI by making reward functions identifiable through active experimentation, though it is incremental in extending prior single-environment results.
The paper tackles the problem of reward function unidentifiability in Inverse Reinforcement Learning (IRL) by proposing an active learning approach where the learner selects multiple environments to observe agent behavior, enabling full theoretical reconstruction of the reward function with a logarithmic number of experiments, and extends this to restricted environments with a near-optimal greedy algorithm validated empirically.
We consider a setting for Inverse Reinforcement Learning (IRL) where the learner is extended with the ability to actively select multiple environments, observing an agent's behavior on each environment. We first demonstrate that if the learner can experiment with any transition dynamics on some fixed set of states and actions, then there exists an algorithm that reconstructs the agent's reward function to the fullest extent theoretically possible, and that requires only a small (logarithmic) number of experiments. We contrast this result to what is known about IRL in single fixed environments, namely that the true reward function is fundamentally unidentifiable. We then extend this setting to the more realistic case where the learner may not select any transition dynamic, but rather is restricted to some fixed set of environments that it may try. We connect the problem of maximizing the information derived from experiments to submodular function maximization and demonstrate that a greedy algorithm is near optimal (up to logarithmic factors). Finally, we empirically validate our algorithm on an environment inspired by behavioral psychology.