COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
This addresses the challenge of measuring compositional generalization in NLP models, which is crucial for developing more robust AI systems, though it is incremental as it focuses on dataset creation and benchmarking.
The authors tackled the problem of evaluating compositional generalization in language models by introducing COGS, a semantic parsing dataset with systematic gaps. They found that while Transformers and LSTMs achieved near-perfect in-distribution accuracy (96-99%), their generalization accuracy was much lower (16-35%) and highly sensitive to random seed (±6-8%).
Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts. To facilitate the evaluation of the compositional abilities of language processing architectures, we introduce COGS, a semantic parsing dataset based on a fragment of English. The evaluation portion of COGS contains multiple systematic gaps that can only be addressed by compositional generalization; these include new combinations of familiar syntactic structures, or new combinations of familiar words and familiar structures. In experiments with Transformers and LSTMs, we found that in-distribution accuracy on the COGS test set was near-perfect (96--99%), but generalization accuracy was substantially lower (16--35%) and showed high sensitivity to random seed ($\pm$6--8%). These findings indicate that contemporary standard NLP models are limited in their compositional generalization capacity, and position COGS as a good way to measure progress.