Preference Elicitation for Offline Reinforcement Learning
This work addresses the problem of designing reward functions and interacting with environments for researchers and practitioners in reinforcement learning, representing an incremental advancement by combining existing frameworks.
The paper tackles the challenge of applying reinforcement learning to real-world problems by bridging offline RL and preference-based RL, proposing Sim-OPRL to acquire preference feedback offline using simulated rollouts, with results including theoretical guarantees and empirical performance demonstrations.
Applying reinforcement learning (RL) to real-world problems is often made challenging by the inability to interact with the environment and the difficulty of designing reward functions. Offline RL addresses the first challenge by considering access to an offline dataset of environment interactions labeled by the reward function. In contrast, Preference-based RL does not assume access to the reward function and learns it from preferences, but typically requires an online interaction with the environment. We bridge the gap between these frameworks by exploring efficient methods for acquiring preference feedback in a fully offline setup. We propose Sim-OPRL, an offline preference-based reinforcement learning algorithm, which leverages a learned environment model to elicit preference feedback on simulated rollouts. Drawing on insights from both the offline RL and the preference-based RL literature, our algorithm employs a pessimistic approach for out-of-distribution data, and an optimistic approach for acquiring informative preferences about the optimal policy. We provide theoretical guarantees regarding the sample complexity of our approach, dependent on how well the offline data covers the optimal policy. Finally, we demonstrate the empirical performance of Sim-OPRL in various environments.