Probing Syntax in Large Language Models: Successes and Remaining Challenges
This work identifies limitations in existing methods for probing syntax in LLMs, providing a controlled benchmark for more accurate evaluation, which is incremental for researchers in NLP and linguistics.
The study analyzed structural probes for reading syntactic structures from large language model activations, finding that they are biased by word proximity and challenged by deep syntactic structures and linguistic interference, while not affected by word predictability.
The syntactic structures of sentences can be readily read-out from the activations of large language models (LLMs). However, the ``structural probes'' that have been developed to reveal this phenomenon are typically evaluated on an indiscriminate set of sentences. Consequently, it remains unclear whether structural and/or statistical factors systematically affect these syntactic representations. To address this issue, we conduct an in-depth analysis of structural probes on three controlled benchmarks. Our results are three-fold. First, structural probes are biased by a superficial property: the closer two words are in a sentence, the more likely structural probes will consider them as syntactically linked. Second, structural probes are challenged by linguistic properties: they poorly represent deep syntactic structures, and get interfered by interacting nouns or ungrammatical verb forms. Third, structural probes do not appear to be affected by the predictability of individual words. Overall, this work sheds light on the current challenges faced by structural probes. Providing a benchmark made of controlled stimuli to better evaluate their performance.