I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling
This work addresses the problem of detecting contradictions in dialogue for improving the consistency of state-of-the-art generative chatbots, which is an incremental improvement for dialogue systems.
This paper introduces the DialoguE COntradiction DEtection task (DECODE) and a new dataset of human-human and human-bot contradictory dialogues to quantify consistency in natural language understanding models. They found that a structured utterance-based Transformer approach is more robust and transferable for contradiction detection than unstructured methods, and their dataset is more effective for supervision than existing NLI data.
To quantify how well natural language understanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We then compare a structured utterance-based approach of using pre-trained Transformer models for contradiction detection with the typical unstructured approach. Results reveal that: (i) our newly collected dataset is notably more effective at providing supervision for the dialogue contradiction detection task than existing NLI data including those aimed to cover the dialogue domain; (ii) the structured utterance-based approach is more robust and transferable on both analysis and out-of-distribution dialogues than its unstructured counterpart. We also show that our best contradiction detection model correlates well with human judgments and further provide evidence for its usage in both automatically evaluating and improving the consistency of state-of-the-art generative chatbots.