Whitening Not Recommended for Classification Tasks in LLMs
This work addresses a practical issue for NLP researchers and practitioners by showing that a common technique is harmful for classification tasks, making it an incremental correction to existing practices.
The paper found that whitening operations, previously claimed to improve embedding quality from Large Language Models (LLMs), actually degrade performance for classification tasks, as shown through extensive experiments with various whitening methods.
Sentence embedding is a cornerstone in NLP. Whitening has been claimed to be an effective operation to improve embedding quality obtained from Large Language Models (LLMs). However, we find that the efficacy of whitening is model-dependent and task-dependent. In particular, whitening degenerates embeddings for classification tasks. The conclusion is supported by extensive experiments. We also explored a variety of whitening operations, including PCA, ZCA, PCA-Cor, ZCA-Cor and Cholesky whitenings. A by-product of our research is embedding evaluation platform for LLMs called SentEval+.