Uppsala NLP at SemEval-2021 Task 2: Multilingual Language Models for Fine-tuning and Feature Extraction in Word-in-Context Disambiguation
This work addresses word sense disambiguation for multilingual NLP applications, but it is incremental as it compares existing models on a specific task.
The paper tackled the problem of multilingual and cross-lingual word-in-context disambiguation by comparing three pre-trained multilingual language models (XLM-RoBERTa, mBERT, mDistilBERT) in fine-tuning and feature extraction setups, finding that fine-tuning outperformed feature extraction and XLM-RoBERTa was better in cross-lingual settings.
We describe the Uppsala NLP submission to SemEval-2021 Task 2 on multilingual and cross-lingual word-in-context disambiguation. We explore the usefulness of three pre-trained multilingual language models, XLM-RoBERTa (XLMR), Multilingual BERT (mBERT) and multilingual distilled BERT (mDistilBERT). We compare these three models in two setups, fine-tuning and as feature extractors. In the second case we also experiment with using dependency-based information. We find that fine-tuning is better than feature extraction. XLMR performs better than mBERT in the cross-lingual setting both with fine-tuning and feature extraction, whereas these two models give a similar performance in the multilingual setting. mDistilBERT performs poorly with fine-tuning but gives similar results to the other models when used as a feature extractor. We submitted our two best systems, fine-tuned with XLMR and mBERT.