Test-Time Alignment via Hypothesis Reweighting
It addresses the challenge of adapting models to diverse user preferences in underspecified scenarios like chatbots, offering a scalable solution for personalization and distribution shifts.
The paper tackles the problem of aligning large pretrained models to test-time user intent in underspecified tasks, proposing HyRe, a method that dynamically reweights ensemble members using a few labeled examples, achieving state-of-the-art accuracy with just five preference pairs per distribution.
Large pretrained models often struggle with underspecified tasks -- situations where the training data does not fully define the desired behavior. For example, chatbots must handle diverse and often conflicting user preferences, requiring adaptability to various user needs. We propose a novel framework to address the general challenge of aligning models to test-time user intent, which is rarely fully specified during training. Our approach involves training an efficient ensemble, i.e., a single neural network with multiple prediction heads, each representing a different function consistent with the training data. Our main contribution is HyRe, a simple adaptation technique that dynamically reweights ensemble members at test time using a small set of labeled examples from the target distribution, which can be labeled in advance or actively queried from a larger unlabeled pool. By leveraging recent advances in scalable ensemble training, our method scales to large pretrained models, with computational costs comparable to fine-tuning a single model. We empirically validate HyRe in several underspecified scenarios, including personalization tasks and settings with distribution shifts. Additionally, with just five preference pairs from each target distribution, the same ensemble adapted via HyRe outperforms the prior state-of-the-art 2B-parameter reward model accuracy across 18 evaluation distributions.