Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy
This addresses the need for efficient, sense-aware embeddings in natural language processing, though it is incremental as it builds on existing embedding methods.
The paper tackled the problem of static word embeddings lacking sensitivity to different word senses by proposing CDES, a method that injects sense information from contextualized embeddings into static ones, achieving comparable performance to state-of-the-art sense embeddings on word sense disambiguation and discrimination benchmarks.
Contextualised word embeddings generated from Neural Language Models (NLMs), such as BERT, represent a word with a vector that considers the semantics of the target word as well its context. On the other hand, static word embeddings such as GloVe represent words by relatively low-dimensional, memory- and compute-efficient vectors but are not sensitive to the different senses of the word. We propose Context Derived Embeddings of Senses (CDES), a method that extracts sense related information from contextualised embeddings and injects it into static embeddings to create sense-specific static embeddings. Experimental results on multiple benchmarks for word sense disambiguation and sense discrimination tasks show that CDES can accurately learn sense-specific static embeddings reporting comparable performance to the current state-of-the-art sense embeddings.