PrOnto: Language Model Evaluations for 859 Languages
This work addresses the problem of evaluating pretrained language models for low-resource languages, though it is incremental as it builds on existing annotation projection techniques.
The authors tackled the scarcity of evaluation datasets for non-English languages by developing a method to automatically create such datasets using New Testament translations, applying it to 859 languages and making them publicly available, with experiments showing the method's efficacy for assessing language model quality.
Evaluation datasets are critical resources for measuring the quality of pretrained language models. However, due to the high cost of dataset annotation, these resources are scarce for most languages other than English, making it difficult to assess the quality of language models. In this work, we present a new method for evaluation dataset construction which enables any language with a New Testament translation to receive a suite of evaluation datasets suitable for pretrained language model evaluation. The method critically involves aligning verses with those in the New Testament portion of English OntoNotes, and then projecting annotations from English to the target language, with no manual annotation required. We apply this method to 1051 New Testament translations in 859 and make them publicly available. Additionally, we conduct experiments which demonstrate the efficacy of our method for creating evaluation tasks which can assess language model quality.