Compositional Generalization with Grounded Language Models
This work addresses the challenge of improving compositional generalization in language models for AI and NLP researchers, but it is incremental as it builds on prior semantic parsing studies.
The paper tackles the problem of compositional generalization in grounded language models by generating natural language questions paired with knowledge graphs to evaluate how well these models learn and generalize from patterns, finding that existing methods struggle with generalization to unseen lengths and novel combinations.
Grounded language models use external sources of information, such as knowledge graphs, to meet some of the general challenges associated with pre-training. By extending previous work on compositional generalization in semantic parsing, we allow for a controlled evaluation of the degree to which these models learn and generalize from patterns in knowledge graphs. We develop a procedure for generating natural language questions paired with knowledge graphs that targets different aspects of compositionality and further avoids grounding the language models in information already encoded implicitly in their weights. We evaluate existing methods for combining language models with knowledge graphs and find them to struggle with generalization to sequences of unseen lengths and to novel combinations of seen base components. While our experimental results provide some insight into the expressive power of these models, we hope our work and released datasets motivate future research on how to better combine language models with structured knowledge representations.