Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation
This addresses a computational bottleneck for researchers and practitioners using instruction-following embeddings, offering a more efficient solution with demonstrated performance gains.
The paper tackles the computational inefficiency of instruction-following text embedding, where existing methods require re-encoding entire corpora for each new instruction, and proposes GSTransform, a lightweight transformation mechanism that adapts pre-computed embeddings in real time, achieving 6~300x speedups while improving embedding quality over state-of-the-art methods.
In this work, we investigate an important task named instruction-following text embedding, which generates dynamic text embeddings that adapt to user instructions, highlighting specific attributes of text. Despite recent advancements, existing approaches suffer from significant computational overhead, as they require re-encoding the entire corpus for each new instruction. To address this challenge, we propose GSTransform, a novel instruction-following text embedding framework based on Guided Space Transformation. Our key observation is that instruction-relevant information is inherently encoded in generic embeddings but remains underutilized. Instead of repeatedly encoding the corpus for each instruction, GSTransform is a lightweight transformation mechanism that adapts pre-computed embeddings in real time to align with user instructions, guided by a small amount of text data with instruction-focused label annotation. We conduct extensive experiments on three instruction-awareness downstream tasks across nine real-world datasets, demonstrating that GSTransform improves instruction-following text embedding quality over state-of-the-art methods while achieving dramatic speedups of 6~300x in real-time processing on large-scale datasets. The source code is available at https://github.com/YingchaojieFeng/GSTransform.