Guiding Symbolic Natural Language Grammar Induction via Transformer-Based Sequence Probabilities
This addresses the challenge of improving symbolic grammar induction for natural language processing, though it appears incremental as it builds on prior research.
The paper tackles the problem of automated syntactic rule learning for natural languages by using transformer language model probabilities to guide symbolic learning processes, demonstrating a proof-of-concept with unsupervised link-grammar induction.
A novel approach to automated learning of syntactic rules governing natural languages is proposed, based on using probabilities assigned to sentences (and potentially longer word sequences) by transformer neural network language models to guide symbolic learning processes like clustering and rule induction. This method exploits the learned linguistic knowledge in transformers, without any reference to their inner representations; hence, the technique is readily adaptable to the continuous appearance of more powerful language models. We show a proof-of-concept example of our proposed technique, using it to guide unsupervised symbolic link-grammar induction methods drawn from our prior research.