Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish
This work addresses the problem of inadequate benchmark datasets for low-resource languages like Turkish, highlighting the need for better quality control, though it is incremental as it focuses on evaluation rather than new dataset creation.
The study evaluated the quality of 17 Turkish benchmark datasets using a framework with six criteria, finding that 70% failed to meet heuristic quality standards and 85% of criteria were unsatisfied, with LLM judges being less effective than humans in cultural and fluency tasks.
The reliance on translated or adapted datasets from English or multilingual resources introduces challenges regarding linguistic and cultural suitability. This study addresses the need for robust and culturally appropriate benchmarks by evaluating the quality of 17 commonly used Turkish benchmark datasets. Using a comprehensive framework that assesses six criteria, both human and LLM-judge annotators provide detailed evaluations to identify dataset strengths and shortcomings. Our results reveal that 70% of the benchmark datasets fail to meet our heuristic quality standards. The correctness of the usage of technical terms is the strongest criterion, but 85% of the criteria are not satisfied in the examined datasets. Although LLM judges demonstrate potential, they are less effective than human annotators, particularly in understanding cultural common sense knowledge and interpreting fluent, unambiguous text. GPT-4o has stronger labeling capabilities for grammatical and technical tasks, while Llama3.3-70B excels at correctness and cultural knowledge evaluation. Our findings emphasize the urgent need for more rigorous quality control in creating and adapting datasets for low-resource languages.