A Set of Quebec-French Corpus of Regional Expressions and Terms
This provides a tool for NLP researchers to evaluate dialect understanding, but it is incremental as it combines existing tasks into a new benchmark.
The authors tackled the problem of dialect understanding by introducing two new benchmark datasets for Quebec French, QFrCoRE and QFrCoRT, containing 4,633 idiomatic phrases and 171 regional words, and demonstrated their reliability in measuring model proficiency with experiments on 94 LLMs.
The tasks of idiom understanding and dialect understanding are both well-established benchmarks in natural language processing. In this paper, we propose combining them, and using regional idioms as a test of dialect understanding. Towards this end, we propose two new benchmark datasets for the Quebec dialect of French: QFrCoRE, which contains 4,633 instances of idiomatic phrases, and QFrCoRT, which comprises 171 regional instances of idiomatic words. We explain how to construct these corpora, so that our methodology can be replicated for other dialects. Our experiments with 94 LLM demonstrate that our regional idiom benchmarks are a reliable tool for measuring a model's proficiency in a specific dialect.