Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changes
This work addresses a fundamental understanding gap in natural language processing for researchers, though it is incremental as it builds on prior studies of embedding geometry.
The study investigated how semantic changes of words are encoded in contextualized word embedding spaces, finding that fine-tuning distributes this information across all dimensions and PCA better represents semantic changes than ICA in the top 10% of axes.
The sense-aware contextualised word embeddings (SCWEs) encode semantic changes of words within the contextualised word embedding (CWE) spaces. Despite the superior performance of SCWEs in contextual/temporal semantic change detection (SCD) benchmarks, it remains unclear as to how the meaning changes are encoded in the embedding space. To study this, we compare pre-trained CWEs and their fine-tuned versions on contextual and temporal semantic change benchmarks under Principal Component Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our experimental results reveal (a) although there exist a smaller number of axes that are specific to semantic changes of words in the pre-trained CWE space, this information gets distributed across all dimensions when fine-tuned, and (b) in contrast to prior work studying the geometry of CWEs, we find that PCA to better represent semantic changes than ICA within the top 10% of axes. These findings encourage the development of more efficient SCD methods with a small number of SCD-aware dimensions. Source code is available at https://github.com/LivNLP/svp-dims .