Hindsight PRIORs for Reward Learning from Human Preferences
This addresses data inefficiency and subpar reward functions in PbRL for robotics and AI applications, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the credit assignment problem in preference-based reinforcement learning by introducing Hindsight PRIOR, a strategy that uses a world model to approximate state importance and guide reward learning, resulting in improved policy performance and reward recovery, such as 20% more reward on MetaWorld and 15% on DMC.
Preference based Reinforcement Learning (PbRL) removes the need to hand specify a reward function by learning a reward from preference feedback over policy behaviors. Current approaches to PbRL do not address the credit assignment problem inherent in determining which parts of a behavior most contributed to a preference, which result in data intensive approaches and subpar reward functions. We address such limitations by introducing a credit assignment strategy (Hindsight PRIOR) that uses a world model to approximate state importance within a trajectory and then guides rewards to be proportional to state importance through an auxiliary predicted return redistribution objective. Incorporating state importance into reward learning improves the speed of policy learning, overall policy performance, and reward recovery on both locomotion and manipulation tasks. For example, Hindsight PRIOR recovers on average significantly (p<0.05) more reward on MetaWorld (20%) and DMC (15%). The performance gains and our ablations demonstrate the benefits even a simple credit assignment strategy can have on reward learning and that state importance in forward dynamics prediction is a strong proxy for a state's contribution to a preference decision. Code repository can be found at https://github.com/apple/ml-rlhf-hindsight-prior.