Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective
This work addresses the need for better benchmarking of LLMs in Chinese commonsense reasoning, though it is incremental as it builds on existing QA datasets and methods.
The authors tackled the understudied evaluation of LLMs in Chinese-language contexts by proposing the CCMOR benchmark to assess multi-hop reasoning with Chinese-specific knowledge, finding that retrieval-augmented generation significantly improves performance by mitigating knowledge gaps.
While Large Language Models (LLMs) have demonstrated advanced reasoning capabilities, their comprehensive evaluation in general Chinese-language contexts remains understudied. To bridge this gap, we propose Chinese Commonsense Multi-hop Reasoning (CCMOR), a novel benchmark designed to evaluate LLMs' ability to integrate Chinese-specific factual knowledge with multi-step logical reasoning. Specifically, we first construct a domain-balanced seed set from existing QA datasets, then develop an LLM-powered pipeline to generate multi-hop questions anchored on factual unit chains. To ensure the quality of resulting dataset, we implement a human-in-the-loop verification system, where domain experts systematically validate and refine the generated questions. Using CCMOR, we evaluate state-of-the-art LLMs, demonstrating persistent limitations in LLMs' ability to process long-tail knowledge and execute knowledge-intensive reasoning. Notably, retrieval-augmented generation substantially mitigates these knowledge gaps, yielding significant performance gains.