A Hierarchical Bayesian model for Inverse RL in Partially-Controlled Environments
This work addresses the challenge of filtering nuisance observations in partially-controlled environments for robots using inverse reinforcement learning, representing an incremental improvement over existing methods.
The paper tackles the problem of robots learning from observations in real-world environments with confounding elements by presenting a hierarchical Bayesian model for inverse reinforcement learning that explicitly models diverse observations, and demonstrates its effectiveness in a simulated robotic sorting domain where it outperforms several comparative methods, second only to having perfect knowledge of the subject's trajectory.
Robots learning from observations in the real world using inverse reinforcement learning (IRL) may encounter objects or agents in the environment, other than the expert, that cause nuisance observations during the demonstration. These confounding elements are typically removed in fully-controlled environments such as virtual simulations or lab settings. When complete removal is impossible the nuisance observations must be filtered out. However, identifying the source of observations when large amounts of observations are made is difficult. To address this, we present a hierarchical Bayesian model that incorporates both the expert's and the confounding elements' observations thereby explicitly modeling the diverse observations a robot may receive. We extend an existing IRL algorithm originally designed to work under partial occlusion of the expert to consider the diverse observations. In a simulated robotic sorting domain containing both occlusion and confounding elements, we demonstrate the model's effectiveness. In particular, our technique outperforms several other comparative methods, second only to having perfect knowledge of the subject's trajectory.