Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
This addresses the problem of inefficient personalization of LLMs for diverse individual human preferences, representing an incremental improvement over existing alignment methods.
The paper tackles the challenge of efficiently aligning large language models with individual human preferences by introducing a method that disentangles preference representation from text generation. The approach achieves aligned quality comparable to or better than PEFT-based methods while reducing additional training time per new individual preference by 80% to 90%.
Aligning Large Language Models (LLMs) with general human preferences has been proved crucial in improving the interaction quality between LLMs and human. However, human values are inherently diverse among different individuals, making it insufficient to align LLMs solely with general preferences. To address this, personalizing LLMs according to individual feedback emerges as a promising solution. Nonetheless, this approach presents challenges in terms of the efficiency of alignment algorithms. In this work, we introduce a flexible paradigm for individual preference alignment. Our method fundamentally improves efficiency by disentangling preference representation from text generation in LLMs. We validate our approach across multiple text generation tasks and demonstrate that it can produce aligned quality as well as or better than PEFT-based methods, while reducing additional training time for each new individual preference by $80\%$ to $90\%$ in comparison with them.