Evaluating Hallucinations in Chinese Large Language Models
This work addresses the issue of hallucinations in Chinese LLMs for researchers and developers, providing a domain-specific benchmark, but it is incremental as it adapts existing evaluation methods to a new language and dataset.
The authors tackled the problem of measuring hallucinations in Chinese large language models by establishing a benchmark called HalluQA, which contains 450 adversarial questions across multiple domains, and found that 18 out of 24 models had non-hallucination rates below 50%, indicating the benchmark is highly challenging.
In this paper, we establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. HalluQA contains 450 meticulously designed adversarial questions, spanning multiple domains, and takes into account Chinese historical culture, customs, and social phenomena. During the construction of HalluQA, we consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated. We conduct extensive experiments on 24 large language models, including ERNIE-Bot, Baichuan2, ChatGLM, Qwen, SparkDesk and etc. Out of the 24 models, 18 achieved non-hallucination rates lower than 50%. This indicates that HalluQA is highly challenging. We analyze the primary types of hallucinations in different types of models and their causes. Additionally, we discuss which types of hallucinations should be prioritized for different types of models.