UDPipe at EvaLatin 2020: Contextualized Embeddings and Treebank Embeddings
This work addresses the evaluation of NLP tools for Latin, an incremental improvement by applying existing methods to a new domain with strong specific gains.
The authors tackled the EvaLatin shared task for NLP tools in Latin, achieving first place in lemmatization and POS tagging in the open modality and best performance in lemmatization and classical POS tagging in the closed modality, with specific gains such as a wide margin in open modality and second place in cross-genre and cross-time settings.
We present our contribution to the EvaLatin shared task, which is the first evaluation campaign devoted to the evaluation of NLP tools for Latin. We submitted a system based on UDPipe 2.0, one of the winners of the CoNLL 2018 Shared Task, The 2018 Shared Task on Extrinsic Parser Evaluation and SIGMORPHON 2019 Shared Task. Our system places first by a wide margin both in lemmatization and POS tagging in the open modality, where additional supervised data is allowed, in which case we utilize all Universal Dependency Latin treebanks. In the closed modality, where only the EvaLatin training data is allowed, our system achieves the best performance in lemmatization and in classical subtask of POS tagging, while reaching second place in cross-genre and cross-time settings. In the ablation experiments, we also evaluate the influence of BERT and XLM-RoBERTa contextualized embeddings, and the treebank encodings of the different flavors of Latin treebanks.