CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
This addresses a bottleneck in preference-based reinforcement learning for improving human-AI alignment, though it is an incremental improvement over existing methods.
The paper tackles the problem of ambiguous human preference labels in preference-based reinforcement learning, which reduces label efficiency, by proposing CLARIFY, a method that learns a trajectory embedding space to select more unambiguous queries; experiments show it outperforms baselines in non-ideal teacher and real human feedback settings.
Preference-based reinforcement learning (PbRL) bypasses explicit reward engineering by inferring reward functions from human preference comparisons, enabling better alignment with human intentions. However, humans often struggle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL's real-world applicability. To address this, we propose an offline PbRL method: Contrastive LeArning for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embedding space that incorporates preference information, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experiments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real human feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings.