Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization
This addresses multi-objective alignment for LLMs when human preferences are unknown, offering a novel a-posteriori optimization approach.
The paper tackles the challenge of aligning large language models with complex, conflicting human preferences by proposing HaM, an algorithm that learns diverse policies to cover the Pareto front through hypervolume maximization. It shows empirical superiority across objectives like harmlessness and helpfulness on various datasets.
Multi-objective alignment from human feedback (MOAHF) in large language models (LLMs) is a challenging problem as human preferences are complex, multifaceted, and often conflicting. Recent works on MOAHF considered a-priori multi-objective optimization (MOO), where human preferences are known at training or inference time. In contrast, when human preferences are unknown or difficult to quantify, a natural approach is to cover the Pareto front by multiple diverse solutions. We propose an algorithm HaM for learning diverse LLM policies that maximizes their hypervolume. This is the first application of a-posteriori MOO to MOAHF. HaM is computationally and space efficient, and empirically superior across objectives such as harmlessness, helpfulness, humor, faithfulness, and hallucination, on various datasets.