Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and English
This work addresses the problem of evaluating syntactic biases in LLMs for linguists and AI researchers, but it is incremental as it builds on existing sentence processing studies.
The study investigated whether monolingual and multilingual large language models (LLMs) exhibit human-like preferences for relative clause attachment ambiguities in Italian and English, finding that models generally fail to capture these preferences accurately.
This paper leverages past sentence processing studies to investigate whether monolingual and multilingual LLMs show human-like preferences when presented with examples of relative clause attachment ambiguities in Italian and English. Furthermore, we test whether these preferences can be modulated by lexical factors (the type of verb/noun in the matrix clause) which have been shown to be tied to subtle constraints on syntactic and semantic relations. Our results overall showcase how LLM behavior varies interestingly across models, but also general failings of these models in correctly capturing human-like preferences. In light of these results, we argue that RC attachment is the ideal benchmark for cross-linguistic investigations of LLMs' linguistic knowledge and biases.