Behavior Preference Regression for Offline Reinforcement Learning
This work addresses the challenge of offline RL for domains like robotics and language modeling, offering a novel method that is incremental in improving policy constraint approaches.
The paper tackles the problem of learning optimal policies from fixed datasets in offline reinforcement learning by introducing Behavior Preference Regression (BPR), which reformulates a paired-sample optimization to fit the Q-function while maximizing behavioral consistency, resulting in state-of-the-art performance on D4RL Locomotion, Antmaze, and V-D4RL datasets.
Offline reinforcement learning (RL) methods aim to learn optimal policies with access only to trajectories in a fixed dataset. Policy constraint methods formulate policy learning as an optimization problem that balances maximizing reward with minimizing deviation from the behavior policy. Closed form solutions to this problem can be derived as weighted behavioral cloning objectives that, in theory, must compute an intractable partition function. Reinforcement learning has gained popularity in language modeling to align models with human preferences; some recent works consider paired completions that are ranked by a preference model following which the likelihood of the preferred completion is directly increased. We adapt this approach of paired comparison. By reformulating the paired-sample optimization problem, we fit the maximum-mode of the Q function while maximizing behavioral consistency of policy actions. This yields our algorithm, Behavior Preference Regression for offline RL (BPR). We empirically evaluate BPR on the widely used D4RL Locomotion and Antmaze datasets, as well as the more challenging V-D4RL suite, which operates in image-based state spaces. BPR demonstrates state-of-the-art performance over all domains. Our on-policy experiments suggest that BPR takes advantage of the stability of on-policy value functions with minimal perceptible performance degradation on Locomotion datasets.