Naturally Occurring Feedback is Common, Extractable and Useful
This addresses the scalability issue in collecting human feedback for language model development, though it is incremental as it builds on existing feedback extraction methods.
The paper tackled the problem of costly and unscalable human feedback collection for language models by proposing to extract naturally occurring feedback from user interactions, finding that 30% of chats contain explicit feedback and that training with extracted feedback improves model alignment and performance over baselines.
Human feedback data is a critical component in developing language models. However, collecting this feedback is costly and ultimately not scalable. Inspired by the way human interlocutors provide spontaneous unsolicited feedback to each other, we propose to extract feedback that users naturally include when interacting with chat models. We manually annotated conversations to confirm the presence of naturally occurring feedback in a standard corpus, finding that as much as 30% of the chats include explicit feedback. Comparing to older datasets, we find that naturally occurring feedback is more prevalent in recent conversation datasets, suggesting that more than ever, naturally occurring feedback can serve as a valuable resource for feedback data. We propose a method for automatically extracting this feedback, and apply it to over 1M conversations to obtain hundreds of thousands of feedback samples. The extracted feedback shows promise: training with it improves over baseline models and enhances model alignment to human preferences.