Symbol Guided Hindsight Priors for Reward Learning from Human Preferences
This work addresses the problem of reducing human feedback needed for reward learning in reinforcement learning, offering a domain-specific incremental improvement.
The paper tackles the challenge of specifying rewards for reinforcement learning agents by introducing the PRIOR framework, which incorporates priors about reward structure and preference feedback into reward learning, reducing required feedback by half and improving reward recovery.
Specifying rewards for reinforcement learned (RL) agents is challenging. Preference-based RL (PbRL) mitigates these challenges by inferring a reward from feedback over sets of trajectories. However, the effectiveness of PbRL is limited by the amount of feedback needed to reliably recover the structure of the target reward. We present the PRIor Over Rewards (PRIOR) framework, which incorporates priors about the structure of the reward function and the preference feedback into the reward learning process. Imposing these priors as soft constraints on the reward learning objective reduces the amount of feedback required by half and improves overall reward recovery. Additionally, we demonstrate that using an abstract state space for the computation of the priors further improves the reward learning and the agent's performance.