ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data
This work addresses multiobjective alignment for large language models to better serve diverse user needs, representing an incremental improvement over existing offline methods.
The paper tackles the problem of aligning large language models with multiple human objectives by introducing ParetoHqD, which uses preference directions and high-quality data near the Pareto front, resulting in superior performance over five baselines on two tasks.
Aligning large language models with multiple human expectations and values is crucial for ensuring that they adequately serve a variety of user needs. To this end, offline multiobjective alignment algorithms such as the Rewards-in-Context algorithm have shown strong performance and efficiency. However, inappropriate preference representations and training with imbalanced reward scores limit the performance of such algorithms. In this work, we introduce ParetoHqD that addresses the above issues by representing human preferences as preference directions in the objective space and regarding data near the Pareto front as ''high-quality'' data. For each preference, ParetoHqD follows a two-stage supervised fine-tuning process, where each stage uses an individual Pareto high-quality training set that best matches its preference direction. The experimental results have demonstrated the superiority of ParetoHqD over five baselines on two multiobjective alignment tasks.