Efficient Sentence Embedding using Discrete Cosine Transform
This work addresses the need for efficient sentence embeddings that capture syntactic structure, offering a practical improvement for natural language processing tasks, though it is incremental relative to existing methods.
The paper tackled the problem of inefficient sentence embeddings by proposing the use of discrete cosine transform (DCT) as an order-preserving compression method, resulting in embeddings that preserve more syntactic information and yield better performance in downstream classification tasks compared to vector averaging.
Vector averaging remains one of the most popular sentence embedding methods in spite of its obvious disregard for syntactic structure. While more complex sequential or convolutional networks potentially yield superior classification performance, the improvements in classification accuracy are typically mediocre compared to the simple vector averaging. As an efficient alternative, we propose the use of discrete cosine transform (DCT) to compress word sequences in an order-preserving manner. The lower order DCT coefficients represent the overall feature patterns in sentences, which results in suitable embeddings for tasks that could benefit from syntactic features. Our results in semantic probing tasks demonstrate that DCT embeddings indeed preserve more syntactic information compared with vector averaging. With practically equivalent complexity, the model yields better overall performance in downstream classification tasks that correlate with syntactic features, which illustrates the capacity of DCT to preserve word order information.