Inference-Time Personalized Alignment with a Few User Preference Queries
This addresses the need for efficient personalization in generative AI without requiring large query sets or explicit text inputs, representing an incremental improvement over existing methods.
The paper tackles the problem of aligning generative models with user preferences by proposing UserAlign, an inference-time method that uses a few pairwise response comparisons to select personalized responses, achieving effective alignment across text and image generation tasks.
We study the problem of aligning a generative model's response with a user's preferences. Recent works have proposed several different formulations for personalized alignment; however, they either require a large amount of user preference queries or require that the preference be explicitly specified as a text input. In this paper, we propose a novel inference-time personalized alignment method, UserAlign, that elicits the user's preferences with a few queries as pairwise response comparisons. In particular, UserAlign builds on the theoretical framework of best-arm identification in logistic bandits and selects a personalized response from a fixed pool of the model's generated responses. The key idea is to consider the user's feedback consistent and noise-free, and incorporate it into the theoretical framework to identify the best response quickly. Experimental results across several tasks, involving personalized text and image generation, showcase the effectiveness of UserAlign in achieving personalized alignment.