How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
This work addresses theoretical gaps in preference alignment methods for LLMs, with implications for improving stability and performance in AI alignment, though it is incremental in building on existing frameworks.
The paper investigates how sampling and reference choices affect the alignment of large language models with human preferences, showing that proper sampling improves ranking guarantees while skewed sampling can cause excessive concentration, and that iterative alignment dynamics can lead to oscillations or entropy collapse under certain conditions.
Standard methods for aligning large language models with human preferences learn from pairwise comparisons among sampled candidate responses and regularize toward a reference policy. Despite their effectiveness, the effects of sampling and reference choices are poorly understood theoretically. We investigate these effects through Identity Preference Optimization, a widely used preference alignment framework, and show that proper instance-dependent sampling can yield stronger ranking guarantees, while skewed on-policy sampling can induce excessive concentration under structured preferences. We then analyze iterative alignment dynamics in which the learned policy feeds back into future sampling and reference policies, reflecting a common practice of model-generated preference data. We prove that these dynamics can exhibit persistent oscillations or entropy collapse for certain parameter choices, and characterize regimes that guarantee stability. Our theoretical insights extend to Direct Preference Optimization, indicating the phenomena we captured are common to a broader class of preference-alignment methods. Experiments on real-world preference data validate our findings.