When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates
This work provides a rigorous benchmark for evaluating cross-lingual semantic reasoning in LLMs, highlighting a critical limitation in resolving form–meaning conflicts for closely related languages.
The authors introduce SemCog Bench, a benchmark of 1,858 Arabic–Hebrew word pairs, and find that LLMs achieve high accuracy on true cognates but performance drops sharply on false friends and loanwords, with sentence-level context providing only modest improvements.
Arabic and Hebrew, as closely related Semitic languages, share a substantial lexicon of true cognates, misleading false friends, and modern loanwords. This overlap poses a challenge for cross-lingual semantic understanding in large language models (LLMs). To evaluate this capability, we introduce SemCog Bench, a curated benchmark of 1,858 Arabic--Hebrew word pairs with sentence-level annotations for cognate identification and semantic disambiguation. We evaluate open-source and commercial LLMs across multiple input representations (raw, diacritized, Romanized, and phonetic) and reveal a critical gap in cross-lingual reasoning. While models achieve high accuracy on true cognates, performance drops sharply on false friends and loanwords, reflecting a strong reliance on surface-form similarity. Furthermore, sentence-level context yields only modest improvements, suggesting that contextual cues alone are insufficient to overcome misleading form-based signals. These findings reveal a fundamental limitation of current LLMs in resolving cross-lingual form--meaning conflicts and establish SemCog Bench as a rigorous benchmark for multilingual semantic reasoning. Our code and data are publicly available.