Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers
This provides a novel, efficient approach for syntactic parsing that leverages pre-trained models, benefiting NLP researchers and practitioners by reducing the need for additional training.
The paper tackles the problem of extracting parse trees from masked language models without training separate parsers by using a chart-based method that scores spans based on contextual distortion from linguistic perturbations, achieving state-of-the-art performance in English and outperforming previous methods in 6 out of 8 multilingual settings.
Recent advancements in pre-trained language models (PLMs) have demonstrated that these models possess some degree of syntactic awareness. To leverage this knowledge, we propose a novel chart-based method for extracting parse trees from masked language models (LMs) without the need to train separate parsers. Our method computes a score for each span based on the distortion of contextual representations resulting from linguistic perturbations. We design a set of perturbations motivated by the linguistic concept of constituency tests, and use these to score each span by aggregating the distortion scores. To produce a parse tree, we use chart parsing to find the tree with the minimum score. Our method consistently outperforms previous state-of-the-art methods on English with masked LMs, and also demonstrates superior performance in a multilingual setting, outperforming the state of the art in 6 out of 8 languages. Notably, although our method does not involve parameter updates or extensive hyperparameter search, its performance can even surpass some unsupervised parsing methods that require fine-tuning. Our analysis highlights that the distortion of contextual representation resulting from syntactic perturbation can serve as an effective indicator of constituency across languages.