High Quality ELMo Embeddings for Seven Less-Resourced Languages
This work addresses the need for better contextual embeddings in specific less-resourced languages, but it is incremental as it applies an existing method to new data.
The authors tackled the problem of low-quality ELMo embeddings for seven less-resourced languages by training new embeddings on larger datasets, showing improvements over baseline FastText embeddings in analogy and NER tasks.
Recent results show that deep neural networks using contextual embeddings significantly outperform non-contextual embeddings on a majority of text classification task. We offer precomputed embeddings from popular contextual ELMo model for seven languages: Croatian, Estonian, Finnish, Latvian, Lithuanian, Slovenian, and Swedish. We demonstrate that the quality of embeddings strongly depends on the size of training set and show that existing publicly available ELMo embeddings for listed languages shall be improved. We train new ELMo embeddings on much larger training sets and show their advantage over baseline non-contextual FastText embeddings. In evaluation, we use two benchmarks, the analogy task and the NER task.