FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data
This addresses the challenge of limited data for personalization in conversational AI, though it is incremental as it builds on existing alignment techniques.
The paper tackles the problem of personalizing LLM-powered conversational assistants to align with individual user preferences when only a small set of annotations is available per user, introducing datasets DnD and ELIP and proposing FaST, which achieves the best overall performance in benchmarks.
LLM-powered conversational assistants are often deployed in a one-size-fits-all manner, which fails to accommodate individual user preferences. Recently, LLM personalization -- tailoring models to align with specific user preferences -- has gained increasing attention as a way to bridge this gap. In this work, we specifically focus on a practical yet challenging setting where only a small set of preference annotations can be collected per user -- a problem we define as Personalized Preference Alignment with Limited Data (PPALLI). To support research in this area, we introduce two datasets -- DnD and ELIP -- and benchmark a variety of alignment techniques on them. We further propose FaST, a highly parameter-efficient approach that leverages high-level features automatically discovered from the data, achieving the best overall performance.