Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitioners
This addresses the challenge for RL practitioners in designing and selecting effective reward functions, offering a practical tool to improve reward alignment, though it is incremental in enhancing existing reward design processes.
The paper tackles the problem of evaluating reward function correctness in reinforcement learning by introducing the Trajectory Alignment Coefficient to measure reward alignment with human preferences. The result shows that using this metric in a user study reduced cognitive workload by 1.5x, was preferred by 82% of users, and increased success rates by 41%.
Reinforcement learning agents are fundamentally limited by the quality of the reward functions they learn from, yet reward design is often overlooked under the assumption that a well-defined reward is readily available. However, in practice, designing rewards is difficult, and even when specified, evaluating their correctness is equally problematic: how do we know if a reward function is correctly specified? In our work, we address these challenges by focusing on reward alignment -- assessing whether a reward function accurately encodes the preferences of a human stakeholder. As a concrete measure of reward alignment, we introduce the Trajectory Alignment Coefficient to quantify the similarity between a human stakeholder's ranking of trajectory distributions and those induced by a given reward function. We show that the Trajectory Alignment Coefficient exhibits desirable properties, such as not requiring access to a ground truth reward, invariance to potential-based reward shaping, and applicability to online RL. Additionally, in an 11 -- person user study of RL practitioners, we found that access to the Trajectory Alignment Coefficient during reward selection led to statistically significant improvements. Compared to relying only on reward functions, our metric reduced cognitive workload by 1.5x, was preferred by 82% of users and increased the success rate of selecting reward functions that produced performant policies by 41%.