AutoReply: Detecting Nonsense in Dialogue Introspectively with Discriminative Replies
This addresses the challenge of improving dialogue coherence in complex, grounded settings like Diplomacy, though it is incremental as it builds on existing introspection concepts.
The paper tackles the problem of detecting nonsense in dialogues by enabling models to introspectively assess their own messages using discriminative replies, achieving performance comparable to fine-tuned supervised models on a Diplomacy dataset.
Existing approaches built separate classifiers to detect nonsense in dialogues. In this paper, we show that without external classifiers, dialogue models can detect errors in their own messages introspectively, by calculating the likelihood of replies that are indicative of poor messages. For example, if an agent believes its partner is likely to respond "I don't understand" to a candidate message, that message may not make sense, so an alternative message should be chosen. We evaluate our approach on a dataset from the game Diplomacy, which contains long dialogues richly grounded in the game state, on which existing models make many errors. We first show that hand-crafted replies can be effective for the task of detecting nonsense in applications as complex as Diplomacy. We then design AutoReply, an algorithm to search for such discriminative replies automatically, given a small number of annotated dialogue examples. We find that AutoReply-generated replies outperform handcrafted replies and perform on par with carefully fine-tuned large supervised models. Results also show that one single reply without much computation overheads can also detect dialogue nonsense reasonably well.