Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?
This work addresses the problem of LLMs' limited human-like language comprehension for researchers and developers, though it is incremental as it builds on existing evaluations of syntactic processing.
The study investigated how large language models (LLMs) resolve syntactic ambiguity in relative clause attachment across six languages, finding that LLMs consistently prefer local attachment and rely heavily on world knowledge biases, unlike humans who show more flexibility.
This study explores how recent large language models (LLMs) navigate relative clause attachment {ambiguity} and use world knowledge biases for disambiguation in six typologically diverse languages: English, Chinese, Japanese, Korean, Russian, and Spanish. We describe the process of creating a novel dataset -- MultiWho -- for fine-grained evaluation of relative clause attachment preferences in ambiguous and unambiguous contexts. Our experiments with three LLMs indicate that, contrary to humans, LLMs consistently exhibit a preference for local attachment, displaying limited responsiveness to syntactic variations or language-specific attachment patterns. Although LLMs performed well in unambiguous cases, they rigidly prioritized world knowledge biases, lacking the flexibility of human language processing. These findings highlight the need for more diverse, pragmatically nuanced multilingual training to improve LLMs' handling of complex structures and human-like comprehension.