NoCoLA: The Norwegian Corpus of Linguistic Acceptability
This provides a domain-specific benchmark for evaluating Norwegian language models, addressing a gap in linguistic acceptability assessment.
The authors tackled the lack of tools to evaluate grammatical understanding in Norwegian language models by presenting two new datasets, NoCoLA_class for supervised classification and NoCoLA_zero for zero-shot evaluation, and conducted a comparative study of existing models.
While there has been a surge of large language models for Norwegian in recent years, we lack any tool to evaluate their understanding of grammaticality. We present two new Norwegian datasets for this task. NoCoLA_class is a supervised binary classification task where the goal is to discriminate between acceptable and non-acceptable sentences. On the other hand, NoCoLA_zero is a purely diagnostic task for evaluating the grammatical judgement of a language model in a completely zero-shot manner, i.e. without any further training. In this paper, we describe both datasets in detail, show how to use them for different flavors of language models, and conduct a comparative study of the existing Norwegian language models.