When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels
This work addresses the challenge of efficiently using implicit feedback for continuous improvement in deployed dialogue systems, representing an incremental advancement.
The paper tackles the problem of improving dialogue agents by converting sparse human feedback into useful training labels, resulting in performance gains through model-corrected replies and leveraging both positive and negative feedback.
Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.