Evaluating Chinese Ambiguity Understanding in Large Language Models
This work addresses the lack of scalable Chinese ambiguity datasets and provides insights into LLMs' handling of Chinese ambiguity, which is important for improving robustness in Chinese NLP.
The authors constructed CHA-Gen, the first Chinese ambiguity dataset grounded in Potential Ambiguity Theory, comprising 5,712 sentences. Evaluating LLMs (e.g., Gemma 3, Qwen 2.5/3) revealed that models struggle with ambiguity detection, show higher uncertainty for ambiguous sentences, and exhibit bias toward dominant interpretations.
Linguistic ambiguity is critical to the robustness of Large Language Models (LLMs), yet existing research focuses mostly on English, with limited attention devoted to Chinese. Existing Chinese ambiguity datasets (e.g., CHAmbi) suffer from poor scalability. Guided by Potential Ambiguity (PA) Theory, we design a semi-automatic pipeline to construct CHA-Gen. It is the first PA Theory-grounded Chinese ambiguity dataset, which comprises 5,712 sentences (2,414 ambiguous, 3,298 unambiguous) across 18 potential ambiguous structures. Evaluating LLMs (e.g. Gemma 3, Qwen 2.5/3 series) via direct querying and machine translation, we find that LLMs struggle with ambiguity detection (improved by CoT prompting). Analysis of Qwen3-32B's CoT rationales reveals three common failure modes: ambiguity blindness, misattribution, and premature resolution. Uncertainty quantification with semantic entropy metric shows higher uncertainty for ambiguous sentences. Moreover, instruction tuning induces overconfidence, whereas Base models better capture semantic diversity. We further observe that models exhibit a bias toward dominant interpretations. Our work provides a scalable approach for Chinese ambiguity corpus and insights into LLMs' ambiguity handling, laying a foundation for enhancing Chinese ambiguity research in LLMs.