Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs
This addresses the challenge of adapting LLMs to diverse and evolving user preferences in real-time, which is crucial for practical applications, though it is an incremental improvement over existing alignment methods.
The paper tackles the problem of aligning large language models with personalized, changing user preferences during test time without retraining, and introduces Amulet, a training-free framework that achieves significant performance improvements across various LLMs, datasets, and preferences while maintaining computational efficiency.
How to align large language models (LLMs) with user preferences from a static general dataset has been frequently studied. However, user preferences are usually personalized, changing, and diverse regarding culture, values, or time. This leads to the problem that the actual user preferences often do not coincide with those trained by the model developers in the practical use of LLMs. Since we cannot collect enough data and retrain for every demand, researching efficient real-time preference adaptation methods based on the backbone LLMs during test time is important. To this end, we introduce Amulet, a novel, training-free framework that formulates the decoding process of every token as a separate online learning problem with the guidance of simple user-provided prompts, thus enabling real-time optimization to satisfy users' personalized preferences. To reduce the computational cost brought by this optimization process for each token, we additionally provide a closed-form solution for each iteration step of the optimization process, thereby reducing the computational time cost to a negligible level. The detailed experimental results demonstrate that Amulet can achieve significant performance improvements in rich settings with combinations of different LLMs, datasets, and user preferences, while maintaining acceptable computational efficiency.