DABERT: Dual Attention Enhanced BERT for Semantic Matching
This work addresses a robustness issue in semantic matching for NLP applications, but it is incremental as it builds upon BERT with enhancements.
The paper tackles the problem of BERT's insufficient ability to capture subtle differences in semantic sentence matching, which can lead to flipped predictions with minor noise, and proposes DABERT, achieving effective results on semantic matching and robustness test datasets.
Transformer-based pre-trained language models such as BERT have achieved remarkable results in Semantic Sentence Matching. However, existing models still suffer from insufficient ability to capture subtle differences. Minor noise like word addition, deletion, and modification of sentences may cause flipped predictions. To alleviate this problem, we propose a novel Dual Attention Enhanced BERT (DABERT) to enhance the ability of BERT to capture fine-grained differences in sentence pairs. DABERT comprises (1) Dual Attention module, which measures soft word matches by introducing a new dual channel alignment mechanism to model affinity and difference attention. (2) Adaptive Fusion module, this module uses attention to learn the aggregation of difference and affinity features, and generates a vector describing the matching details of sentence pairs. We conduct extensive experiments on well-studied semantic matching and robustness test datasets, and the experimental results show the effectiveness of our proposed method.