Compressing Word Embeddings Using Syllables
This work addresses the need for more efficient word embeddings in NLP, particularly for resource-constrained applications, but it is incremental as it builds on existing subword embedding methods.
The paper tackles the problem of compressing word embeddings by using syllable embeddings as subword units instead of n-grams, achieving models that are 20-30 times smaller for English with 80% performance retention and 15 times smaller for Dutch with 70% retention, though less accurate than n-gram baselines.
This work examines the possibility of using syllable embeddings, instead of the often used $n$-gram embeddings, as subword embeddings. We investigate this for two languages: English and Dutch. To this end, we also translated two standard English word embedding evaluation datasets, WordSim353 and SemEval-2017, to Dutch. Furthermore, we provide the research community with data sets of syllabic decompositions for both languages. We compare our approach to full word and $n$-gram embeddings. Compared to full word embeddings, we obtain English models that are 20 to 30 times smaller while retaining 80% of the performance. For Dutch, models are 15 times smaller for 70% performance retention. Although less accurate than the $n$-gram baseline we used, our models can be trained in a matter of minutes, as opposed to hours for the $n$-gram approach. We identify a path toward upgrading performance in future work. All code is made publicly available, as well as our collected English and Dutch syllabic decompositions and Dutch evaluation set translations.