Polyglot Contextual Representations Improve Crosslingual Transfer
This work addresses the challenge of crosslingual transfer for NLP tasks, offering a method to share representations across languages, but it is incremental as it builds on existing multilingual and contextual representation techniques.
The authors tackled the problem of crosslingual transfer by introducing Rosita, a method to produce multilingual contextual word representations through training a single language model on multiple languages, and demonstrated improvements in tasks like dependency parsing, semantic role labeling, and named entity recognition for language pairs such as English/Arabic and English/Chinese.
We introduce Rosita, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages. Our method combines the advantages of contextual word representations with those of multilingual representation learning. We produce language models from dissimilar language pairs (English/Arabic and English/Chinese) and use them in dependency parsing, semantic role labeling, and named entity recognition, with comparisons to monolingual and non-contextual variants. Our results provide further evidence for the benefits of polyglot learning, in which representations are shared across multiple languages.