Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation
This addresses the need for efficient semantic retrieval in NLP by compressing sentence representations, though it is incremental as it builds on existing distillation and projection techniques.
The paper tackles the problem of learning compact sentence embeddings by proposing Homomorphic Projective Distillation (HPD), which achieves a 2.7-4.5 point gain on semantic textual similarity tasks and improves retrieval speed by 8.2x and memory usage by 8.0x compared to large models.
How to learn highly compact yet effective sentence representation? Pre-trained language models have been effective in many NLP tasks. However, these models are often huge and produce large sentence embeddings. Moreover, there is a big performance gap between large and small models. In this paper, we propose Homomorphic Projective Distillation (HPD) to learn compressed sentence embeddings. Our method augments a small Transformer encoder model with learnable projection layers to produce compact representations while mimicking a large pre-trained language model to retain the sentence representation quality. We evaluate our method with different model sizes on both semantic textual similarity (STS) and semantic retrieval (SR) tasks. Experiments show that our method achieves 2.7-4.5 points performance gain on STS tasks compared with previous best representations of the same size. In SR tasks, our method improves retrieval speed (8.2$\times$) and memory usage (8.0$\times$) compared with state-of-the-art large models.