Quantification and object perception in Multimodal Large Language Models deviate from human linguistic cognition
This work addresses the problem of understanding MLLMs' semantic and pragmatic limitations for researchers in AI and linguistics, though it is incremental as it builds on known performance issues.
The paper investigated how Multimodal Large Language Models (MLLMs) handle quantification, a difficult linguistic phenomenon, by comparing them to human cognition across three cross-linguistic features, finding clear differences in tasks that test in vivo vs. in silico representations.
Quantification has been proven to be a particularly difficult linguistic phenomenon for (Multimodal) Large Language Models (MLLMs). However, given that quantification interfaces with the logic, pragmatic, and numerical domains, the exact reasons for the poor performance are still unclear. This papers looks at three key features of human quantification shared cross-linguistically that have remained so far unexplored in the (M)LLM literature: the ordering of quantifiers into scales, the ranges of use and prototypicality, and the biases inherent in the human approximate number system. The aim is to determine how these features are encoded in the models' architecture, how they may differ from humans, and whether the results are affected by the type of model and language under investigation. We find that there are clear differences between humans and MLLMs with respect to these features across various tasks that tap into the representation of quantification in vivo vs. in silico. This work, thus, paves the way for addressing the nature of MLLMs as semantic and pragmatic agents, while the cross-linguistic lens can elucidate whether their abilities are robust and stable across different languages.