Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE
This work addresses the gap in assessing LLMs for ancient language understanding, which is incremental as it provides a new benchmark for a specific domain.
The authors tackled the problem of evaluating large language models' comprehension of ancient Chinese by introducing ACLUE, a benchmark with 15 tasks, and found that ChatGLM2 performed best with an average score of 37.4%.
Large language models (LLMs) have showcased remarkable capabilities in understanding and generating language. However, their ability in comprehending ancient languages, particularly ancient Chinese, remains largely unexplored. To bridge this gap, we present ACLUE, an evaluation benchmark designed to assess the capability of language models in comprehending ancient Chinese. ACLUE consists of 15 tasks cover a range of skills, spanning phonetic, lexical, syntactic, semantic, inference and knowledge. Through the evaluation of eight state-of-the-art LLMs, we observed a noticeable disparity in their performance between modern Chinese and ancient Chinese. Among the assessed models, ChatGLM2 demonstrates the most remarkable performance, achieving an average score of 37.4%. We have made our code and data public available.