MAVIS: Multi-Objective Alignment via Value-Guided Inference-Time Search
This addresses the need for flexible and efficient multi-objective alignment in LLMs for diverse applications, offering a computationally cheaper alternative to fine-tuning.
The paper tackles the problem of aligning large language models to multiple, often conflicting objectives like helpfulness and harmlessness without expensive fine-tuning, by introducing MAVIS, a lightweight inference-time framework that uses value models to adjust outputs, achieving performance close to models fine-tuned for specific preferences.
Large Language Models (LLMs) are increasingly deployed across diverse applications that demand balancing multiple, often conflicting, objectives -- such as helpfulness, harmlessness, or humor. Aligning outputs to user-specific preferences in such multi-objective settings typically requires fine-tuning models for each objective or preference configuration, which is computationally expensive and inflexible. We introduce MAVIS -- Multi-Objective Alignment via Value-Guided Inference-Time Search -- a lightweight inference-time alignment framework that enables dynamic control over LLM behavior without modifying the base model's weights. MAVIS trains a set of small value models, each corresponding to a distinct objective. At inference time, these value models are combined using user-specified weights to produce a tilting function that adjusts the base model's output distribution toward desired trade-offs. The value models are trained using a simple iterative algorithm that ensures monotonic improvement of the KL-regularized policy. We show empirically that MAVIS outperforms baselines that fine-tune per-objective models and combine them post hoc, and even approaches the performance of the idealized setting where models are fine-tuned for a user's exact preferences.