Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models
This work addresses the need for efficient, language-agnostic syntactic parsing in multilingual NLP, though it is incremental as it builds on existing paradigms for parse extraction from PLMs.
The authors tackled the problem of extracting constituency parses from pre-trained language models without separate training, proposing a chart-based method with top-K ensemble that achieves performance superior or comparable to unsupervised PCFGs across nine languages.
As it has been unveiled that pre-trained language models (PLMs) are to some extent capable of recognizing syntactic concepts in natural language, much effort has been made to develop a method for extracting complete (binary) parses from PLMs without training separate parsers. We improve upon this paradigm by proposing a novel chart-based method and an effective top-K ensemble technique. Moreover, we demonstrate that we can broaden the scope of application of the approach into multilingual settings. Specifically, we show that by applying our method on multilingual PLMs, it becomes possible to induce non-trivial parses for sentences from nine languages in an integrated and language-agnostic manner, attaining performance superior or comparable to that of unsupervised PCFGs. We also verify that our approach is robust to cross-lingual transfer. Finally, we provide analyses on the inner workings of our method. For instance, we discover universal attention heads which are consistently sensitive to syntactic information irrespective of the input language.