SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures
This addresses the issue of unpleasant user experiences and reduced feedback in conversational AI, though it is incremental as it builds on existing fine-tuning approaches.
The paper tackles the problem of conversational models reacting poorly to safety feedback by proposing SaFeRDialogues, a dataset of 10k dialogues with graceful responses, and shows that fine-tuning on it makes conversations significantly more civil according to human raters.
Current open-domain conversational models can easily be made to talk in inadequate ways. Online learning from conversational feedback given by the conversation partner is a promising avenue for a model to improve and adapt, so as to generate fewer of these safety failures. However, current state-of-the-art models tend to react to feedback with defensive or oblivious responses. This makes for an unpleasant experience and may discourage conversation partners from giving feedback in the future. This work proposes SaFeRDialogues, a task and dataset of graceful responses to conversational feedback about safety failures. We collect a dataset of 10k dialogues demonstrating safety failures, feedback signaling them, and a response acknowledging the feedback. We show how fine-tuning on this dataset results in conversations that human raters deem considerably more likely to lead to a civil conversation, without sacrificing engagingness or general conversational ability.