Offline Constrained RLHF with Multiple Preference Oracles
This addresses the problem of balancing performance with safety or fairness in offline RLHF for applications requiring constraint satisfaction, representing a novel theoretical contribution rather than an incremental improvement.
The paper tackles offline constrained reinforcement learning from human feedback with multiple preference oracles, aiming to maximize target population utility while ensuring minimum protected group welfare. It proposes a dual-only algorithm that guarantees high-probability constraint satisfaction and provides the first finite-sample performance guarantees for this problem.
We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.