Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
This addresses the challenge of learning complex, history-dependent rewards in AI and robotics, representing an incremental advance over prior work focused on Markovian or binary non-Markovian rewards.
The paper tackles the problem of inferring non-Markovian reward functions, such as reward machines, from expert behavior in inverse reinforcement learning, proposing a Bayesian framework that adapts demonstrations and uses simulated annealing, and demonstrates favorable performance compared to an existing method for binary rewards.
Inverse reinforcement learning (IRL) is the problem of inferring a reward function from expert behavior. There are several approaches to IRL, but most are designed to learn a Markovian reward. However, a reward function might be non-Markovian, depending on more than just the current state, such as a reward machine (RM). Although there has been recent work on inferring RMs, it assumes access to the reward signal, absent in IRL. We propose a Bayesian IRL (BIRL) framework for inferring RMs directly from expert behavior, requiring significant changes to the standard framework. We define a new reward space, adapt the expert demonstration to include history, show how to compute the reward posterior, and propose a novel modification to simulated annealing to maximize this posterior. We demonstrate that our method performs well when optimizing according to its inferred reward and compares favorably to an existing method that learns exclusively binary non-Markovian rewards.