Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
This addresses a critical issue for developers and users of LLMs by enhancing alignment under distribution shifts, though it is incremental as it builds on existing optimization methods.
The paper tackles the problem of distribution shifts undermining preference alignment in large language models (LLMs) when using synthetic data, proposing a distribution-aware optimization framework that improves alignment by minimizing worst-case loss over target distribution regions, resulting in better response generation.
Preference alignment methods are increasingly critical for steering large language models (LLMs) to generate outputs consistent with human values. While recent approaches often rely on synthetic data generated by LLMs for scalability and cost-efficiency reasons, this reliance can introduce distribution shifts that undermine the nuanced representation of human preferences needed for desirable outputs. In this paper, we propose a novel distribution-aware optimization framework that improves preference alignment despite such shifts. Our approach first leverages well-learned classifiers to assign a calibration value to each training sample, quantifying its alignment with the target human-preferred distribution. These values are then incorporated into a robust optimization objective that minimizes the worst-case loss over regions of the data space most relevant to human preferences. By explicitly focusing optimization on the target distribution, our approach mitigates the impact of distributional mismatch and improves the generation of responses that better reflect intended values.