Modeling Order in Neural Word Embeddings at Scale
This addresses a bottleneck in NLP for better semantic and syntactic modeling, with incremental improvements in scale and efficiency.
The paper tackles the problem of capturing morphological information and word order in neural language models, achieving 85.8% on a word-analogy task with a 58% error margin improvement and training a 160-billion-parameter network overnight on 3 CPUs, 14x larger than prior networks.
Natural Language Processing (NLP) systems commonly leverage bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. The resulting word-level distributed representations often ignore morphological information, though character-level embeddings have proven valuable to NLP tasks. We propose a new neural language model incorporating both word order and character order in its embedding. The model produces several vector spaces with meaningful substructure, as evidenced by its performance of 85.8% on a recent word-analogy task, exceeding best published syntactic word-analogy scores by a 58% error margin. Furthermore, the model includes several parallel training methods, most notably allowing a skip-gram network with 160 billion parameters to be trained overnight on 3 multi-core CPUs, 14x larger than the previous largest neural network.