Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
This addresses the problem of inefficient and user-burdened multi-objective alignment for LLM practitioners, representing an incremental improvement by automating preference weight specification.
The paper tackles the challenge of aligning large language models with varying human preferences across multiple objectives by proposing PRO, a framework that uses a lightweight adapter to automatically infer prompt-specific preference weights, achieving superior performance over existing methods in experiments.
While Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, aligning these models with varying human preferences across multiple objectives remains a significant challenge in practical deployments. Existing multi-objective alignment methods rely on manually specified preference weights, which not only burden users with difficult preference specification tasks but also lead to suboptimal training efficiency due to exploration of irrelevant preference combinations. To alleviate these issues, we propose a novel framework named PRO, i.e., PReference Orchestrator, which features a lightweight preference adapter that automatically infers prompt-specific preference weights during both training and deployment phases. Specifically, the adapter automatically learns appropriate preference weights for each prompt by training on normalized reward scores from multiple reward models for preferred responses, which inherently reflect effective preference balances across objectives. Additionally, We provide theoretical analysis proving that our prompt-aware preference mechanism achieves superior performance compared to fixed preference weights in multi-objective alignment scenarios. Extensive experiments across multiple tasks demonstrate the effectiveness of our method over existing multi-objective alignment approaches.