User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal
This work addresses the challenge of noisy implicit feedback for improving deployed language models, but it is incremental as it builds on existing datasets and methods.
The study analyzed implicit user feedback from interaction logs with language models to understand its occurrence and utility as a learning signal, finding that incorporating feedback content improved performance on short human-designed questions but not on longer, complex ones.
Once language models (LMs) are deployed, they can interact with users long-term, ideally evolving based on their feedback. Asking for direct user feedback can be disruptive; thus, we study harvesting implicit user feedback from user-LM interaction logs. We study two user-LM interaction datasets (WildChat and LMSYS). First, we analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs. Second, we study harvesting learning signals from such implicit user feedback. Specifically, we study whether incorporating the contents of user feedback (e.g., user wanted clarification), in addition to the polarity of the feedback, can improve the model performance. We observe mixed results, showing this helps in short human-designed questions (MTBench) but not on longer and more complex questions (WildBench). Together, we provide an in-depth study of implicit user feedback, showing its potential and limitations.