Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
This addresses the problem of computational inefficiency in inference-time alignment for LLMs, offering a potentially incremental improvement in multi-criteria alignment.
The paper tackles the computational expense of aligning large language models to multiple human preferences at inference time by proposing inference-aware meta-alignment (IAMA), which trains a base model to be efficiently aligned using different algorithms, achieving this with limited computational budget.
Aligning large language models (LLMs) to diverse human preferences is fundamentally challenging since criteria can often conflict with each other. Inference-time alignment methods have recently gained popularity as they allow LLMs to be aligned to multiple criteria via different alignment algorithms at inference time. However, inference-time alignment is computationally expensive since it often requires multiple forward passes of the base model. In this work, we propose inference-aware meta-alignment (IAMA), a novel approach that enables LLMs to be aligned to multiple criteria with limited computational budget at inference time. IAMA trains a base model such that it can be effectively aligned to multiple tasks via different inference-time alignment algorithms. To solve the non-linear optimization problems involved in IAMA, we propose non-linear GRPO, which provably converges to the optimal solution in the space of probability measures.