SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics
This addresses the challenge of evaluating compositional semantics in NLP for researchers, but it is incremental as it builds on existing testbeds and methods.
The authors tackled the problem of whether deep neural networks can capture compositional meanings in natural language by proposing SyGNS, a testbed for systematic generalization, and found that models like Transformers and GRUs generalize only to unseen combinations similar in form to training instances, with performance improving for simpler meaning representations.
Recently, deep neural networks (DNNs) have achieved great success in semantically challenging NLP tasks, yet it remains unclear whether DNN models can capture compositional meanings, those aspects of meaning that have been long studied in formal semantics. To investigate this issue, we propose a Systematic Generalization testbed based on Natural language Semantics (SyGNS), whose challenge is to map natural language sentences to multiple forms of scoped meaning representations, designed to account for various semantic phenomena. Using SyGNS, we test whether neural networks can systematically parse sentences involving novel combinations of logical expressions such as quantifiers and negation. Experiments show that Transformer and GRU models can generalize to unseen combinations of quantifiers, negations, and modifiers that are similar to given training instances in form, but not to the others. We also find that the generalization performance to unseen combinations is better when the form of meaning representations is simpler. The data and code for SyGNS are publicly available at https://github.com/verypluming/SyGNS.