Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledge
This addresses a key limitation in PLMs for tasks requiring semantic similarity, such as entity linking and matching, though it is incremental as it builds on existing adapter-based methods.
The paper tackled the problem of Pre-trained Language Models lacking contextual synonym knowledge for similarity-oriented tasks like entity linking and matching, and proposed PICSO, a framework that injects such knowledge via an entity-aware Adapter, resulting in dramatic performance improvements over original PLMs and other models on four tasks, with benefits also shown on GLUE benchmarks.
Contextual synonym knowledge is crucial for those similarity-oriented tasks whose core challenge lies in capturing semantic similarity between entities in their contexts, such as entity linking and entity matching. However, most Pre-trained Language Models (PLMs) lack synonym knowledge due to inherent limitations of their pre-training objectives such as masked language modeling (MLM). Existing works which inject synonym knowledge into PLMs often suffer from two severe problems: (i) Neglecting the ambiguity of synonyms, and (ii) Undermining semantic understanding of original PLMs, which is caused by inconsistency between the exact semantic similarity of the synonyms and the broad conceptual relevance learned from the original corpus. To address these issues, we propose PICSO, a flexible framework that supports the injection of contextual synonym knowledge from multiple domains into PLMs via a novel entity-aware Adapter which focuses on the semantics of the entities (synonyms) in the contexts. Meanwhile, PICSO stores the synonym knowledge in additional parameters of the Adapter structure, which prevents it from corrupting the semantic understanding of the original PLM. Extensive experiments demonstrate that PICSO can dramatically outperform the original PLMs and the other knowledge and synonym injection models on four different similarity-oriented tasks. In addition, experiments on GLUE prove that PICSO also benefits general natural language understanding tasks. Codes and data will be public.