Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models
This work provides a new evaluation benchmark for developers to assess Chinese factuality abilities in LLMs, but it is incremental as it adapts an existing benchmark to a new language.
The authors introduced Chinese SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality of large language models in answering short questions, and used it to assess existing models, though no specific performance numbers were provided.
New LLM evaluation benchmarks are important to align with the rapid development of Large Language Models (LLMs). In this work, we present Chinese SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions, and Chinese SimpleQA mainly has five properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6 major topics with 99 diverse subtopics. Second, we conduct a comprehensive quality control process to achieve high-quality questions and answers, where the reference answers are static and cannot be changed over time. Third, following SimpleQA, the questions and answers are very short, and the grading process is easy-to-evaluate based on OpenAI API. Based on Chinese SimpleQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to better understand the Chinese factuality abilities of their models and facilitate the growth of foundation models.