User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction
This addresses the issue of users needing to restate preferences across sessions in conversational agents, though it is incremental as it builds on existing retrieval-augmented methods.
The paper tackles the problem of LLM-based personal assistants lacking persistent user models by proposing VARS, a framework that uses long-term and short-term vectors to bias retrieval scoring for personalization, resulting in improved interaction efficiency and reduced timeout rates and user effort, while matching baseline task success.
Large language models are increasingly used as personal assistants, yet most lack a persistent user model, forcing users to repeatedly restate preferences across sessions. We propose Vector-Adapted Retrieval Scoring (VARS), a pipeline-agnostic, frozen-backbone framework that represents each user with long-term and short-term vectors in a shared preference space and uses these vectors to bias retrieval scoring over structured preference memory. The vectors are updated online from weak scalar rewards from users' feedback, enabling personalization without per-user fine-tuning. We evaluate on \textsc{MultiSessionCollab}, an online multi-session collaboration benchmark with rich user preference profiles, across math and code tasks. Under frozen backbones, the main benefit of user-aware retrieval is improved interaction efficiency rather than large gains in raw task accuracy: our full VARS agent achieves the strongest overall performance, matches a strong Reflection baseline in task success, and reduces timeout rate and user effort. The learned long-term vectors also align with cross-user preference overlap, while short-term vectors capture session-specific adaptation, supporting the interpretability of the dual-vector design. Code, model, and data are available at https://github.com/YurenHao0426/VARS.