A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
This work addresses the challenge of limited linguistic resources for mid-resource languages, offering a practical solution that enhances NLP tasks like tagging and parsing, though it is incremental as it applies an existing method to new data sources.
The authors tackled the problem of generating high-quality contextualized word embeddings for mid-resource languages by training monolingual ELMo models on the noisy OSCAR corpus from Common Crawl, resulting in embeddings that equal or improve the state of the art in part-of-speech tagging and parsing for five languages, outperforming both Wikipedia-based monolingual and multilingual BERT embeddings.
We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.