Aging Memories Generate More Fluent Dialogue Responses with Memory Augmented Neural Networks
This addresses a specific bottleneck in memory-augmented neural networks for dialogue systems, representing an incremental improvement with a novel regularization technique.
The paper tackles the problem of conventional Memory Networks storing highly correlated vectors that cause overfitting when memory size increases, by proposing a novel regularization mechanism called memory dropout that samples from redundant memories and ages them to increase overwriting probability. This approach achieves state-of-the-art response generation on the Stanford Multi-Turn Dialogue and Cambridge Restaurant datasets.
Memory Networks have emerged as effective models to incorporate Knowledge Bases (KB) into neural networks. By storing KB embeddings into a memory component, these models can learn meaningful representations that are grounded to external knowledge. However, as the memory unit becomes full, the oldest memories are replaced by newer representations. In this paper, we question this approach and provide experimental evidence that conventional Memory Networks store highly correlated vectors during training. While increasing the memory size mitigates this problem, this also leads to overfitting as the memory stores a large number of training latent representations. To address these issues, we propose a novel regularization mechanism named memory dropout which 1) Samples a single latent vector from the distribution of redundant memories. 2) Ages redundant memories thus increasing their probability of overwriting them during training. This fully differentiable technique allows us to achieve state-of-the-art response generation in the Stanford Multi-Turn Dialogue and Cambridge Restaurant datasets.