FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models
This addresses the problem of evaluating Chinese LLMs for researchers and developers, though it is incremental as it builds on existing benchmarking efforts.
The paper tackled the challenge of assessing fundamental knowledge in Chinese large language models by introducing FoundaBench, a benchmark with 3354 multiple-choice questions, and found that models pre-trained on Chinese corpora performed better, revealing a disparity between reasoning and memory recall.
In the burgeoning field of large language models (LLMs), the assessment of fundamental knowledge remains a critical challenge, particularly for models tailored to Chinese language and culture. This paper introduces FoundaBench, a pioneering benchmark designed to rigorously evaluate the fundamental knowledge capabilities of Chinese LLMs. FoundaBench encompasses a diverse array of 3354 multiple-choice questions across common sense and K-12 educational subjects, meticulously curated to reflect the breadth and depth of everyday and academic knowledge. We present an extensive evaluation of 12 state-of-the-art LLMs using FoundaBench, employing both traditional assessment methods and our CircularEval protocol to mitigate potential biases in model responses. Our results highlight the superior performance of models pre-trained on Chinese corpora, and reveal a significant disparity between models' reasoning and memory recall capabilities. The insights gleaned from FoundaBench evaluations set a new standard for understanding the fundamental knowledge of LLMs, providing a robust framework for future advancements in the field.