SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation
This addresses a key challenge in scientific document understanding for researchers and NLP practitioners, though it appears incremental as it builds on existing pre-trained models.
The paper tackles the problem of acronym disambiguation in scientific documents by proposing SimCLAD, a contrastive learning framework that improves phrase-level representations, resulting in outperforming state-of-the-art methods on English scientific domain benchmarks.
Acronym disambiguation means finding the correct meaning of an ambiguous acronym from the dictionary in a given sentence, which is one of the key points for scientific document understanding (SDU@AAAI-22). Recently, many attempts have tried to solve this problem via fine-tuning the pre-trained masked language models (MLMs) in order to obtain a better acronym representation. However, the acronym meaning is varied under different contexts, whose corresponding phrase representation mapped in different directions lacks discrimination in the entire vector space. Thus, the original representations of the pre-trained MLMs are not ideal for the acronym disambiguation task. In this paper, we propose a Simple framework for Contrastive Learning of Acronym Disambiguation (SimCLAD) method to better understand the acronym meanings. Specifically, we design a continual contrastive pre-training method that enhances the pre-trained model's generalization ability by learning the phrase-level contrastive distributions between true meaning and ambiguous phrases. The results on the acronym disambiguation of the scientific domain in English show that the proposed method outperforms all other competitive state-of-the-art (SOTA) methods.