A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machines
This addresses reward specification problems in reinforcement learning for complex tasks, though it appears incremental as an augmentation of existing reward machine formalism.
The paper tackles the problem of misspecified rewards degrading reinforcement learning performance by proposing symbolic reward machines that incorporate high-level task knowledge, with experimental results showing learned reward machines significantly improve training efficiency and generalize well across different task configurations.
A misspecified reward can degrade sample efficiency and induce undesired behaviors in reinforcement learning (RL) problems. We propose symbolic reward machines for incorporating high-level task knowledge when specifying the reward signals. Symbolic reward machines augment existing reward machine formalism by allowing transitions to carry predicates and symbolic reward outputs. This formalism lends itself well to inverse reinforcement learning, whereby the key challenge is determining appropriate assignments to the symbolic values from a few expert demonstrations. We propose a hierarchical Bayesian approach for inferring the most likely assignments such that the concretized reward machine can discriminate expert demonstrated trajectories from other trajectories with high accuracy. Experimental results show that learned reward machines can significantly improve training efficiency for complex RL tasks and generalize well across different task environment configurations.