Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems
This work addresses the challenge of designing reward functions for real-world decision-making, offering theoretical insights that could benefit researchers and practitioners in reinforcement learning, though it is incremental as it builds on existing empirical successes.
The paper tackles the problem of learning reward functions from human feedback in offline contextual bandits, showing how human bias and uncertainty affect theoretical guarantees and providing a theoretical explanation for the empirical success of preference-based methods.
For a real-world decision-making problem, the reward function often needs to be engineered or learned. A popular approach is to utilize human feedback to learn a reward function for training. The most straightforward way to do so is to ask humans to provide ratings for state-action pairs on an absolute scale and take these ratings as reward samples directly. Another popular way is to ask humans to rank a small set of state-action pairs by preference and learn a reward function from these preference data. Recently, preference-based methods have demonstrated substantial success in empirical applications such as InstructGPT. In this work, we develop a theoretical comparison between these human feedback approaches in offline contextual bandits and show how human bias and uncertainty in feedback modelings can affect the theoretical guarantees of these approaches. Through this, our results seek to provide a theoretical explanation for the empirical successes of preference-based methods from a modeling perspective.