Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks
This work addresses the problem of embedding size efficiency for researchers and practitioners in NLP, showing that significant dimension reduction has minimal impact, which is incremental but offers practical insights for model optimization.
The paper investigates the effect of randomly removing up to 50% of dimensions in text embeddings, finding that it results in only a minor performance drop of less than 10% across 6 state-of-the-art encoders and 26 downstream tasks in retrieval and classification, and extends this observation to generative tasks with large language models.
In this paper, we study the surprising impact that truncating text embeddings has on downstream performance. We consistently observe across 6 state-of-the-art text encoders and 26 downstream tasks, that randomly removing up to 50% of embedding dimensions results in only a minor drop in performance, less than 10%, in retrieval and classification tasks. Given the benefits of using smaller-sized embeddings, as well as the potential insights about text encoding, we study this phenomenon and find that, contrary to what is suggested in prior work, this is not the result of an ineffective use of representation space. Instead, we find that a large number of uniformly distributed dimensions actually cause an increase in performance when removed. This would explain why, on average, removing a large number of embedding dimensions results in a marginal drop in performance. We make similar observations when truncating the embeddings used by large language models to make next-token predictions on generative tasks, suggesting that this phenomenon is not isolated to classification or retrieval tasks.