Aligning LLMs with Domain Invariant Reward Models
This addresses the problem of domain adaptation for LLM alignment, enabling preference learning in data-scarce target domains, though it is incremental as it builds on existing reward modeling techniques.
The paper tackles the challenge of aligning large language models with human preferences in domains lacking preference data by leveraging feedback from simpler source domains, achieving accuracy improvements from 0.621 to 0.661 in cross-lingual transfer and up to 0.920 in few-shot-to-full transfer.
Aligning large language models (LLMs) to human preferences is challenging in domains where preference data is unavailable. We address the problem of learning reward models for such target domains by leveraging feedback collected from simpler source domains, where human preferences are easier to obtain. Our key insight is that, while domains may differ significantly, human preferences convey \emph{domain-agnostic} concepts that can be effectively captured by a reward model. We propose \method, a framework that trains domain-invariant reward models by optimizing a dual loss: a domain loss that minimizes the divergence between source and target distribution, and a source loss that optimizes preferences on the source domain. We show \method is a general approach that we evaluate and analyze across 4 distinct settings: (1) Cross-lingual transfer (accuracy: $0.621 \rightarrow 0.661$), (2) Clean-to-noisy (accuracy: $0.671 \rightarrow 0.703$), (3) Few-shot-to-full transfer (accuracy: $0.845 \rightarrow 0.920$), and (4) Simple-to-complex tasks transfer (correlation: $0.508 \rightarrow 0.556$). Our code, models and data are available at \url{https://github.com/portal-cornell/dial}.