Relational Memory Augmented Language Models
This work addresses the challenge of generating more coherent and logical text for natural language processing applications, though it appears incremental as it builds on existing memory-augmented and knowledge graph integration methods.
The paper tackled the problem of improving text generation by conditioning autoregressive language models on knowledge graphs, resulting in better performance on perplexity and bits per character across WikiText-103, WMT19, and enwik8 datasets.
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model with a knowledge graph for a more coherent and logical generation.