Advances in Pre-Training Distributed Word Representations
This provides improved word representations for NLP practitioners, though it is incremental as it builds on existing methods.
The paper tackles the problem of training high-quality word vector representations for NLP applications by combining known techniques rarely used together, resulting in new pre-trained models that outperform state-of-the-art by a large margin on multiple tasks.
Many Natural Language Processing applications nowadays rely on pre-trained word representations estimated from large text corpora such as news collections, Wikipedia and Web Crawl. In this paper, we show how to train high-quality word vector representations by using a combination of known tricks that are however rarely used together. The main result of our work is the new set of publicly available pre-trained models that outperform the current state of the art by a large margin on a number of tasks.