Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?
This work addresses the problem of understanding cognitive mechanisms in AI for researchers in NLP and cognitive science, but it is incremental as it applies known experimental designs to models.
The study investigated whether neural language models compute abstract sameness relations similarly to humans, finding that infants outperform all tested pre-trained language models in this cognitive task.
In recent years, deep neural language models have made strong progress in various NLP tasks. This work explores one facet of the question whether state-of-the-art NLP models exhibit elementary mechanisms known from human cognition. The exploration is focused on a relatively primitive mechanism for which there is a lot of evidence from various psycholinguistic experiments with infants. The computation of "abstract sameness relations" is assumed to play an important role in human language acquisition and processing, especially in learning more complex grammar rules. In order to investigate this mechanism in BERT and other pre-trained language models (PLMs), the experiment designs from studies with infants were taken as the starting point. On this basis, we designed experimental settings in which each element from the original studies was mapped to a component of language models. Even though the task in our experiments was relatively simple, the results suggest that the cognitive faculty of computing abstract sameness relations is stronger in infants than in all investigated PLMs.