COLE: a Comprehensive Benchmark for French Language Understanding Evaluation
This addresses the problem of limited evaluation resources for French NLU, providing a public benchmark to foster progress, though it is incremental as it adapts existing benchmark concepts to a specific language.
The authors tackled the lack of comprehensive evaluation for French Natural Language Understanding by introducing COLE, a benchmark with 23 diverse tasks, and found a significant performance gap between closed- and open-weights models while identifying key challenging frontiers for LLMs.
To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a broad range of NLU capabilities, including sentiment analysis, paraphrase detection, grammatical judgment, and reasoning, with a particular focus on linguistic phenomena relevant to the French language. We benchmark 94 large language models (LLM), providing an extensive analysis of the current state of French NLU. Our results highlight a significant performance gap between closed- and open-weights models and identify key challenging frontiers for current LLMs, such as zero-shot extractive question-answering (QA), fine-grained word sense disambiguation, and understanding of regional language variations. We release COLE as a public resource to foster further progress in French language modelling.