Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing
This research addresses a problem for NLP researchers and developers by revealing critical limitations in LLMs' ability to process cross-lingual ambiguities, which is incremental as it builds on existing studies of bilingual processing in humans and models.
The study investigated how multilingual Large Language Models (LLMs) handle bilingual word processing, specifically focusing on cognates, non-cognates, and interlingual homographs. The results showed that LLMs perform well with cognates and non-cognates in isolation but struggle significantly with disambiguating interlingual homographs, often performing below random baselines, indicating a reliance on orthographic similarities over semantic understanding.
Bilingual lexical processing is shaped by the complex interplay of phonological, orthographic, and semantic features of two languages within an integrated mental lexicon. In humans, this is evident in the ease with which cognate words - words similar in both orthographic form and meaning (e.g., blind, meaning "sightless" in both English and German) - are processed, compared to the challenges posed by interlingual homographs, which share orthographic form but differ in meaning (e.g., gift, meaning "present" in English but "poison" in German). We investigate how multilingual Large Language Models (LLMs) handle such phenomena, focusing on English-Spanish, English-French, and English-German cognates, non-cognate, and interlingual homographs. Specifically, we evaluate their ability to disambiguate meanings and make semantic judgments, both when these word types are presented in isolation or within sentence contexts. Our findings reveal that while certain LLMs demonstrate strong performance in recognizing cognates and non-cognates in isolation, they exhibit significant difficulty in disambiguating interlingual homographs, often performing below random baselines. This suggests LLMs tend to rely heavily on orthographic similarities rather than semantic understanding when interpreting interlingual homographs. Further, we find LLMs exhibit difficulty in retrieving word meanings, with performance in isolative disambiguation tasks having no correlation with semantic understanding. Finally, we study how the LLM processes interlingual homographs in incongruent sentences. We find models to opt for different strategies in understanding English and non-English homographs, highlighting a lack of a unified approach to handling cross-lingual ambiguities.