Generalizing Word Embeddings using Bag of Subwords
This addresses the limitation of fixed vocabularies in NLP for researchers and practitioners, though it is incremental as it builds on subword methods.
The paper tackles the problem of generalizing pre-trained word embeddings to handle rare or unseen words by modeling words as bags of character n-grams, achieving state-of-the-art performance in English word similarity and multilingual part-of-speech tagging tasks.
We approach the problem of generalizing pre-trained word embeddings beyond fixed-size vocabularies without using additional contextual information. We propose a subword-level word vector generation model that views words as bags of character $n$-grams. The model is simple, fast to train and provides good vectors for rare or unseen words. Experiments show that our model achieves state-of-the-art performances in English word similarity task and in joint prediction of part-of-speech tag and morphosyntactic attributes in 23 languages, suggesting our model's ability in capturing the relationship between words' textual representations and their embeddings.