Systematic Generalization with Edge Transformers
This addresses the problem of systematic generalization for natural language processing researchers, offering a novel hybrid approach that is incremental in combining existing paradigms.
The paper tackles the challenge of systematic generalization in natural language understanding by proposing Edge Transformer, a model combining Transformers and rule-based symbolic AI, which outperforms baselines on compositional generalization benchmarks in relational reasoning, semantic parsing, and dependency parsing.
Recent research suggests that systematic generalization in natural language understanding remains a challenge for state-of-the-art neural models such as Transformers and Graph Neural Networks. To tackle this challenge, we propose Edge Transformer, a new model that combines inspiration from Transformers and rule-based symbolic AI. The first key idea in Edge Transformers is to associate vector states with every edge, that is, with every pair of input nodes -- as opposed to just every node, as it is done in the Transformer model. The second major innovation is a triangular attention mechanism that updates edge representations in a way that is inspired by unification from logic programming. We evaluate Edge Transformer on compositional generalization benchmarks in relational reasoning, semantic parsing, and dependency parsing. In all three settings, the Edge Transformer outperforms Relation-aware, Universal and classical Transformer baselines.