From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
For researchers in reinforcement learning, this work introduces a new paradigm for offline preference-based RL that improves feedback efficiency, though it is an incremental step combining existing ideas.
This paper rethinks offline preference-based reinforcement learning by connecting it with reward-free representation learning, proposing a framework that first learns latent successor-measure representations from reward-free data, then uses contrastive search and fine-tuning with preferences. The method achieves superior preference efficiency over existing offline PbRL baselines.
Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL.