Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines
For researchers and practitioners working on LLM alignment, this provides a structured framework to understand and improve data construction pipelines, but it is a survey with no new experimental results.
This survey reframes alignment tuning for LLMs as a data pipeline design problem, decomposing it into three stages and organizing existing methods into a unified taxonomy. It identifies design trade-offs and principles, and outlines open challenges.
Much of the alignment tuning literature is organized around optimization objectives, while the construction of alignment data is often treated implicitly. In this survey, we adopt a data centric perspective and reframe alignment tuning as a pipeline design problem. We decompose alignment data construction into three interacting stages, response synthesis, preference evaluation, and preference instantiation, and use this framework to organize existing alignment methods into a unified taxonomy. Through this lens, we identify recurring design trade-offs and failure modes observed across prior alignment methods, and distill a set of high level principles that clarify how pipeline design choices influence the resulting optimization signal. Finally, we outline open challenges for alignment data pipelines, including prompt-level alignment, agentic settings, and alignment under evolving objectives.