Dynamic Contextualized Word Embeddings
This work addresses semantic variability in NLP tasks, but it is incremental as it builds on prior contextualized and dynamic embedding methods.
The authors tackled the problem of static word embeddings failing to capture word meaning variability across contexts by introducing dynamic contextualized word embeddings that model time and social space, demonstrating their potential through analyses on four English datasets.
Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts. Building on prior work on contextualized and dynamic word embeddings, we introduce dynamic contextualized word embeddings that represent words as a function of both linguistic and extralinguistic context. Based on a pretrained language model (PLM), dynamic contextualized word embeddings model time and social space jointly, which makes them attractive for a range of NLP tasks involving semantic variability. We highlight potential application scenarios by means of qualitative and quantitative analyses on four English datasets.