Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
This addresses the interpretability of pre-trained language models for researchers in NLP, though it is incremental as it builds on existing efforts to understand model inner workings.
The paper tackles the problem of understanding whether pre-trained language models capture syntactic constituency, proposing a method to extract constituency trees without training and finding that these models outperform other approaches in demarcating adverb phrases.
With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.