Combining Reward Information from Multiple Sources
This addresses the challenge of reward misspecification in reinforcement learning for agents, though it appears incremental as it builds on existing inverse reward design concepts.
The paper tackles the problem of combining conflicting reward functions from multiple sources when observation models are misspecified, proposing the Multitask Inverse Reward Design (MIRD) algorithm to balance conservatism and informativeness, with empirical results showing it strikes a good balance in a toy environment.
Given two sources of evidence about a latent variable, one can combine the information from both by multiplying the likelihoods of each piece of evidence. However, when one or both of the observation models are misspecified, the distributions will conflict. We study this problem in the setting with two conflicting reward functions learned from different sources. In such a setting, we would like to retreat to a broader distribution over reward functions, in order to mitigate the effects of misspecification. We assume that an agent will maximize expected reward given this distribution over reward functions, and identify four desiderata for this setting. We propose a novel algorithm, Multitask Inverse Reward Design (MIRD), and compare it to a range of simple baselines. While all methods must trade off between conservatism and informativeness, through a combination of theory and empirical results on a toy environment, we find that MIRD and its variant MIRD-IF strike a good balance between the two.