Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform
This addresses the need for efficient embedding storage and processing in NLP applications, though it is incremental as it adapts an existing mathematical tool to a new domain.
The paper tackles the problem of compressing word and sentence embeddings by applying Discrete Wavelet Transform (DWT), achieving dimensionality reductions of 50-93% with minimal performance loss in semantic similarity tasks and improved accuracy in downstream tasks.
Wavelet transforms, a powerful mathematical tool, have been widely used in different domains, including Signal and Image processing, to unravel intricate patterns, enhance data representation, and extract meaningful features from data. Tangible results from their application suggest that Wavelet transforms can be applied to NLP capturing a variety of linguistic and semantic properties. In this paper, we empirically leverage the application of Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We aim to showcase the capabilities of DWT in analyzing embedding representations at different levels of resolution and compressing them while maintaining their overall quality. We assess the effectiveness of DWT embeddings on semantic similarity tasks to show how DWT can be used to consolidate important semantic information in an embedding vector. We show the efficacy of the proposed paradigm using different embedding models, including large language models, on downstream tasks. Our results show that DWT can reduce the dimensionality of embeddings by 50-93% with almost no change in performance for semantic similarity tasks, while achieving superior accuracy in most downstream tasks. Our findings pave the way for applying DWT to improve NLP applications.