Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
This work addresses the limitation of fixed preference weights in multi-objective alignment for LLMs, offering a more flexible and effective training approach.
Meta-Aligner introduces a bi-level meta-learning framework for multi-objective LLM alignment that dynamically adjusts preference weights during training, outperforming static-weight methods on multiple benchmarks.
Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.