Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
This addresses the alignment tax issue for AI developers and users by enabling more flexible and optimal trade-offs in multi-objective alignment, though it is incremental as it builds on existing alignment techniques.
The paper tackles the problem of multi-objective alignment in AI, where improving one objective like harmlessness can degrade others like helpfulness, by introducing controllable preference optimization (CPO) that specifies preference scores to guide model responses, resulting in aligned models that match various preferences and surpass baseline methods in mitigating the alignment tax.
Alignment in artificial intelligence pursues the consistency between model responses and human preferences as well as values. In practice, the multifaceted nature of human preferences inadvertently introduces what is known as the "alignment tax" -a compromise where enhancements in alignment within one objective (e.g.,harmlessness) can diminish performance in others (e.g.,helpfulness). However, existing alignment techniques are mostly unidirectional, leading to suboptimal trade-offs and poor flexibility over various objectives. To navigate this challenge, we argue the prominence of grounding LLMs with evident preferences. We introduce controllable preference optimization (CPO), which explicitly specifies preference scores for different objectives, thereby guiding the model to generate responses that meet the requirements. Our experimental analysis reveals that the aligned models can provide responses that match various preferences among the "3H" (helpfulness, honesty, harmlessness) desiderata. Furthermore, by introducing diverse data and alignment goals, we surpass baseline methods in aligning with single objectives, hence mitigating the impact of the alignment tax and achieving improvements in multi-objective alignment.