The emergent algebraic structure of RNNs and embeddings in NLP
This work addresses foundational structure in NLP models, offering theoretical insights that could influence model design, though it appears incremental as it builds on existing RNN and embedding concepts.
The authors investigated the algebraic and geometric properties of GRUs and word embeddings in NLP, finding that words embed in a Lie group and RNNs form nonlinear representations, leading to proposals for new recurrent networks and embedding schemes.
We examine the algebraic and geometric properties of a uni-directional GRU and word embeddings trained end-to-end on a text classification task. A hyperparameter search over word embedding dimension, GRU hidden dimension, and a linear combination of the GRU outputs is performed. We conclude that words naturally embed themselves in a Lie group and that RNNs form a nonlinear representation of the group. Appealing to these results, we propose a novel class of recurrent-like neural networks and a word embedding scheme.