Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models
This work provides insights into how LLMs capture linguistic knowledge, which is important for researchers in natural language processing and linguistics, though it is incremental in applying minimal pairs to LLMs.
The researchers used linguistic minimal pairs to analyze the internal linguistic representations of Large Language Models (LLMs), finding that linguistic similarity is influenced by training data exposure, aligns with fine-grained theoretical categories, and shows limited cross-lingual alignment across 100+ LLMs and 150k minimal pairs in three languages.
We introduce a novel analysis that leverages linguistic minimal pairs to probe the internal linguistic representations of Large Language Models (LLMs). By measuring the similarity between LLM activation differences across minimal pairs, we quantify the and gain insight into the linguistic knowledge captured by LLMs. Our large-scale experiments, spanning 100+ LLMs and 150k minimal pairs in three languages, reveal properties of linguistic similarity from four key aspects: consistency across LLMs, relation to theoretical categorizations, dependency to semantic context, and cross-lingual alignment of relevant phenomena. Our findings suggest that 1) linguistic similarity is significantly influenced by training data exposure, leading to higher cross-LLM agreement in higher-resource languages. 2) Linguistic similarity strongly aligns with fine-grained theoretical linguistic categories but weakly with broader ones. 3) Linguistic similarity shows a weak correlation with semantic similarity, showing its context-dependent nature. 4) LLMs exhibit limited cross-lingual alignment in their understanding of relevant linguistic phenomena. This work demonstrates the potential of minimal pairs as a window into the neural representations of language in LLMs, shedding light on the relationship between LLMs and linguistic theory. Codes and data are available at https://github.com/ChenDelong1999/Linguistic-Similarity