Understanding Learned Reward Functions
This work addresses the critical problem of verifying whether learned reward functions accurately capture user preferences, which is important for ensuring the safety and reliability of RL agents in real-world applications.
This paper investigates techniques for interpreting learned reward functions in reinforcement learning, which are often necessary when procedural specification is not possible. The authors apply saliency methods to identify failure modes and predict robustness, finding that learned reward functions often implement surprising algorithms reliant on environmental contingencies.
In many real-world tasks, it is not possible to procedurally specify an RL agent's reward function. In such cases, a reward function must instead be learned from interacting with and observing humans. However, current techniques for reward learning may fail to produce reward functions which accurately reflect user preferences. Absent significant advances in reward learning, it is thus important to be able to audit learned reward functions to verify whether they truly capture user preferences. In this paper, we investigate techniques for interpreting learned reward functions. In particular, we apply saliency methods to identify failure modes and predict the robustness of reward functions. We find that learned reward functions often implement surprising algorithms that rely on contingent aspects of the environment. We also discover that existing interpretability techniques often attend to irrelevant changes in reward output, suggesting that reward interpretability may need significantly different methods from policy interpretability.