OmniEduBench: A Comprehensive Chinese Benchmark for Evaluating Large Language Models in Education
This addresses the problem of limited and non-diverse benchmarks for evaluating LLMs in educational contexts, particularly for Chinese applications, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the lack of comprehensive benchmarks for evaluating large language models (LLMs) in Chinese education by introducing OmniEduBench, a dataset with 24.602K question-answer pairs across knowledge and cultivation dimensions, revealing that only Gemini-2.5 Pro exceeded 60% accuracy in knowledge and the best model trailed human intelligence by nearly 30% in cultivation.
With the rapid development of large language models (LLMs), various LLM-based works have been widely applied in educational fields. However, most existing LLMs and their benchmarks focus primarily on the knowledge dimension, largely neglecting the evaluation of cultivation capabilities that are essential for real-world educational scenarios. Additionally, current benchmarks are often limited to a single subject or question type, lacking sufficient diversity. This issue is particularly prominent within the Chinese context. To address this gap, we introduce OmniEduBench, a comprehensive Chinese educational benchmark. OmniEduBench consists of 24.602K high-quality question-answer pairs. The data is meticulously divided into two core dimensions: the knowledge dimension and the cultivation dimension, which contain 18.121K and 6.481K entries, respectively. Each dimension is further subdivided into 6 fine-grained categories, covering a total of 61 different subjects (41 in the knowledge and 20 in the cultivation). Furthermore, the dataset features a rich variety of question formats, including 11 common exam question types, providing a solid foundation for comprehensively evaluating LLMs' capabilities in education. Extensive experiments on 11 mainstream open-source and closed-source LLMs reveal a clear performance gap. In the knowledge dimension, only Gemini-2.5 Pro surpassed 60\% accuracy, while in the cultivation dimension, the best-performing model, QWQ, still trailed human intelligence by nearly 30\%. These results highlight the substantial room for improvement and underscore the challenges of applying LLMs in education.