Grammatical information in BERT sentence embeddings as two-dimensional arrays
This work addresses the challenge of accessing rule-like grammatical information in language model representations, which is incremental but aids in developing few-shot learning approaches for NLP.
The paper tackled the problem of extracting grammatical information from BERT sentence embeddings by reshaping them into two-dimensional arrays, showing that this allows learning architectures to detect subject-verb agreement patterns with improved performance on complex test data using simpler training data.
Sentence embeddings induced with various transformer architectures encode much semantic and syntactic information in a distributed manner in a one-dimensional array. We investigate whether specific grammatical information can be accessed in these distributed representations. Using data from a task developed to test rule-like generalizations, our experiments on detecting subject-verb agreement yield several promising results. First, we show that while the usual sentence representations encoded as one-dimensional arrays do not easily support extraction of rule-like regularities, a two-dimensional reshaping of these vectors allows various learning architectures to access such information. Next, we show that various architectures can detect patterns in these two-dimensional reshaped sentence embeddings and successfully learn a model based on smaller amounts of simpler training data, which performs well on more complex test data. This indicates that current sentence embeddings contain information that is regularly distributed, and which can be captured when the embeddings are reshaped into higher dimensional arrays. Our results cast light on representations produced by language models and help move towards developing few-shot learning approaches.