Representation Degeneration Problem in Training Natural Language Generation Models
This addresses a critical bottleneck in training large-scale natural language generation models, improving representation quality and performance, though it is incremental as it builds on existing regularization techniques.
The paper tackles the representation degeneration problem in neural language generation models, where word embeddings collapse into a narrow cone during training, limiting their power; their proposed regularization method mitigates this issue and achieves better performance on language modeling and machine translation tasks.
We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.