LLMs Struggle with NLI for Perfect Aspect: A Cross-Linguistic Study in Chinese and Japanese
This addresses a problem for NLP researchers and developers working on cross-linguistic temporal semantics, but it is incremental as it focuses on specific languages and aspects.
The study tackled the challenge of Natural Language Inference (NLI) for the perfect aspect in Chinese and Japanese, where grammatical forms differ from English, and found that advanced LLMs struggle with temporal inference, particularly in detecting subtle tense and reference-time shifts, using a dataset of 1,350 pairs per language.
Unlike English, which uses distinct forms (e.g., had, has, will have) to mark the perfect aspect across tenses, Chinese and Japanese lack separate grammatical forms for tense within the perfect aspect, which complicates Natural Language Inference (NLI). Focusing on the perfect aspect in these languages, we construct a linguistically motivated, template-based NLI dataset (1,350 pairs per language). Experiments reveal that even advanced LLMs struggle with temporal inference, particularly in detecting subtle tense and reference-time shifts. These findings highlight model limitations and underscore the need for cross-linguistic evaluation in temporal semantics. Our dataset is available at https://github.com/Lujie2001/CrossNLI.