Multilingual Culture-Independent Word Analogy Datasets
This provides a more standardized tool for evaluating text embeddings across multiple languages, though it is incremental as it builds on existing analogy tasks.
The authors tackled the lack of culturally independent benchmarks for word embeddings by creating multilingual word analogy datasets in nine languages, including cross-lingual versions, and provided initial evaluation results using fastText embeddings.
In text processing, deep neural networks mostly use word embeddings as an input. Embeddings have to ensure that relations between words are reflected through distances in a high-dimensional numeric space. To compare the quality of different text embeddings, typically, we use benchmark datasets. We present a collection of such datasets for the word analogy task in nine languages: Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish. We redesigned the original monolingual analogy task to be much more culturally independent and also constructed cross-lingual analogy datasets for the involved languages. We present basic statistics of the created datasets and their initial evaluation using fastText embeddings.