Probing Multilingual Language Models for Discourse
This provides a comprehensive evaluation platform for multilingual discourse understanding, though the findings are incremental extensions of existing multilingual model analysis.
The paper investigates how well multilingual language models transfer discourse-level knowledge across languages through systematic evaluation on 5 discourse tasks covering 22 languages. They found XLM-RoBERTa models performed best as both monolingual models and in zero-shot settings, with model distillation hurting cross-lingual transfer while language dissimilarity had modest effects.
Pre-trained multilingual language models have become an important building block in multilingual natural language processing. In the present paper, we investigate a range of such models to find out how well they transfer discourse-level knowledge across languages. This is done with a systematic evaluation on a broader set of discourse-level tasks than has been previously been assembled. We find that the XLM-RoBERTa family of models consistently show the best performance, by simultaneously being good monolingual models and degrading relatively little in a zero-shot setting. Our results also indicate that model distillation may hurt the ability of cross-lingual transfer of sentence representations, while language dissimilarity at most has a modest effect. We hope that our test suite, covering 5 tasks with a total of 22 languages in 10 distinct families, will serve as a useful evaluation platform for multilingual performance at and beyond the sentence level.