Static Word Embeddings for Sentence Semantic Representation
This work addresses the need for computationally efficient sentence representations, though it is incremental as it builds on existing embedding methods.
The paper tackles the problem of creating efficient static word embeddings for sentence semantic representation by optimizing pre-trained embeddings with sentence-level PCA and additional training, and shows that their model substantially outperforms existing static models and even surpasses a basic Sentence Transformer on a benchmark.
We propose new static word embeddings optimised for sentence semantic representation. We first extract word embeddings from a pre-trained Sentence Transformer, and improve them with sentence-level principal component analysis, followed by either knowledge distillation or contrastive learning. During inference, we represent sentences by simply averaging word embeddings, which requires little computational cost. We evaluate models on both monolingual and cross-lingual tasks and show that our model substantially outperforms existing static models on sentence semantic tasks, and even surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. Lastly, we perform a variety of analyses and show that our method successfully removes word embedding components that are not highly relevant to sentence semantics, and adjusts the vector norms based on the influence of words on sentence semantics.