Injecting Wiktionary to improve token-level contextual representations using contrastive learning
This work addresses lexical semantics tasks for NLP researchers, offering an incremental improvement by leveraging Wiktionary for better contextual representations.
The paper tackled the problem of improving contextual word embeddings for lexical semantics by injecting Wiktionary as a supervision source and using contrastive learning, achieving new state-of-the-art results on the Word-In-Context task and proposing new test sets with substantial improvements.
While static word embeddings are blind to context, for lexical semantics tasks context is rather too present in contextual word embeddings, vectors of same-meaning occurrences being too different (Ethayarajh, 2019). Fine-tuning pre-trained language models (PLMs) using contrastive learning was proposed, leveraging automatically self-augmented examples (Liu et al., 2021b). In this paper, we investigate how to inject a lexicon as an alternative source of supervision, using the English Wiktionary. We also test how dimensionality reduction impacts the resulting contextual word embeddings. We evaluate our approach on the Word-In-Context (WiC) task, in the unsupervised setting (not using the training set). We achieve new SoTA result on the original WiC test set. We also propose two new WiC test sets for which we show that our fine-tuning method achieves substantial improvements. We also observe improvements, although modest, for the semantic frame induction task. Although we experimented on English to allow comparison with related work, our method is adaptable to the many languages for which large Wiktionaries exist.