Coherent Dialogue with Attention-based Language Models
This work addresses coherence in dialogue generation for applications like chatbots, though it is incremental as it builds on existing attention and RNN frameworks.
The authors tackled the problem of generating coherent dialogue continuations by introducing a dynamic attention mechanism in RNN-based models, which adapts attention scope over conversation history, and achieved significant improvements over state-of-the-art methods on metrics like diversity and human evaluation across two datasets.
We model coherent conversation continuation via RNN-based dialogue models equipped with a dynamic attention mechanism. Our attention-RNN language model dynamically increases the scope of attention on the history as the conversation continues, as opposed to standard attention (or alignment) models with a fixed input scope in a sequence-to-sequence model. This allows each generated word to be associated with the most relevant words in its corresponding conversation history. We evaluate the model on two popular dialogue datasets, the open-domain MovieTriples dataset and the closed-domain Ubuntu Troubleshoot dataset, and achieve significant improvements over the state-of-the-art and baselines on several metrics, including complementary diversity-based metrics, human evaluation, and qualitative visualizations. We also show that a vanilla RNN with dynamic attention outperforms more complex memory models (e.g., LSTM and GRU) by allowing for flexible, long-distance memory. We promote further coherence via topic modeling-based reranking.