Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
This addresses the costly and unstable alignment of large foundation models for AI safety, though it appears incremental as an adaptation of existing methods.
The paper tackles the problem of multi-objective alignment of foundation models with human preferences by introducing Rewards-in-Context (RiC), which uses supervised fine-tuning and dynamic inference-time adjustment to achieve alignment with only around 10% of the GPU hours compared to multi-objective RL baselines.
We consider the problem of multi-objective alignment of foundation models with human preferences, which is a critical step towards helpful and harmless AI systems. However, it is generally costly and unstable to fine-tune large foundation models using reinforcement learning (RL), and the multi-dimensionality, heterogeneity, and conflicting nature of human preferences further complicate the alignment process. In this paper, we introduce Rewards-in-Context (RiC), which conditions the response of a foundation model on multiple rewards in its prompt context and applies supervised fine-tuning for alignment. The salient features of RiC are simplicity and adaptivity, as it only requires supervised fine-tuning of a single foundation model and supports dynamic adjustment for user preferences during inference time. Inspired by the analytical solution of an abstracted convex optimization problem, our dynamic inference-time adjustment method approaches the Pareto-optimal solution for multiple objectives. Empirical evidence demonstrates the efficacy of our method in aligning both Large Language Models (LLMs) and diffusion models to accommodate diverse rewards with only around 10% GPU hours compared with multi-objective RL baseline.