Soft Preference Optimization: Aligning Language Models to Expert Distributions
This addresses the problem of aligning generative models like LLMs for AI safety and usability, though it appears incremental as it builds on existing preference optimization techniques.
The paper tackles aligning language models with human preferences by proposing Soft Preference Optimization (SPO), which directly optimizes outputs over preference data without a reward model, resulting in a method that is simpler, more computationally efficient, and more precise in alignment.
We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.