Structural generalization is hard for sequence-to-sequence models
This addresses a key limitation in NLP for researchers and practitioners, showing that seq2seq models fail at compositional generalization, which is incremental as it builds on prior work but extends it to broader tasks.
The paper demonstrates that sequence-to-sequence models struggle with structural generalization across NLP tasks like semantic parsing, syntactic parsing, and text-to-text tasks, achieving low accuracy on unseen linguistic structures, while neurosymbolic models with built-in linguistic knowledge often overcome this limitation.
Sequence-to-sequence (seq2seq) models have been successful across many NLP tasks, including ones that require predicting linguistic structure. However, recent work on compositional generalization has shown that seq2seq models achieve very low accuracy in generalizing to linguistic structures that were not seen in training. We present new evidence that this is a general limitation of seq2seq models that is present not just in semantic parsing, but also in syntactic parsing and in text-to-text tasks, and that this limitation can often be overcome by neurosymbolic models that have linguistic knowledge built in. We further report on some experiments that give initial answers on the reasons for these limitations.