Towards Equipping Transformer with the Ability of Systematic Compositionality
This addresses a key limitation in language models for AI researchers, though it is incremental as it builds on BERT.
The paper tackles the problem of Transformers lacking systematic compositionality, proposing a compositionality-aware Transformer (CAT) with novel pre-training tasks that outperforms baselines on compositionality tasks while maintaining performance on standard language understanding tasks.
One of the key factors in language productivity and human cognition is the ability of systematic compositionality, which refers to understanding composed unseen examples of seen primitives. However, recent evidence reveals that the Transformers have difficulty generalizing the composed context based on the seen primitives. To this end, we take the first step to propose a compositionality-aware Transformer called CAT and two novel pre-training tasks to facilitate systematic compositionality. We tentatively provide a successful implementation of a multi-layer CAT on the basis of the especially popular BERT. The experimental results demonstrate that CAT outperforms baselines on compositionality-aware tasks with minimal impact on the effectiveness on standardized language understanding tasks.