Unsupervised Human Preference Learning
This work tackles the problem of efficient and effective personalization of large language models for individual users, which is an incremental improvement over existing methods.
This paper addresses the challenge of personalizing large language models (LLMs) for individual users, which is difficult due to the complexity of human preferences and small personal datasets. The authors propose using a small parameter model as a "preference agent" to generate natural language rules that guide a larger, pre-trained LLM, achieving personalization without fine-tuning the large model. Experiments on email and article datasets show their technique significantly outperforms baseline personalization methods.
Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content due to their lack of individual user preference information. Existing methods, such as in-context learning and parameter-efficient fine-tuning, fall short in capturing the complexity of human preferences, especially given the small, personal datasets individuals possess. In this paper, we propose a novel approach utilizing small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization. Our method involves a small, local "steering wheel" model that directs the outputs of a much larger foundation model, producing content tailored to an individual's preferences while leveraging the extensive knowledge and capabilities of the large model. Importantly, this personalization is achieved without the need to fine-tune the large model. Experimental results on email and article datasets, demonstrate that our technique significantly outperforms baseline personalization methods. By allowing foundation models to adapt to individual preferences in a data and compute-efficient manner, our approach paves the way for highly personalized language model applications.