Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement
This addresses the limitation of encoder-only LLM embeddings for NLP practitioners by offering a new paradigm, though it is incremental in refining existing generative approaches.
The paper tackles the problem of LLM-based embeddings by introducing GIRCSE, a framework that uses autoregressive generation and iterative contrastive refinement to capture latent semantics, outperforming baselines on MTEB and instruction-following tasks with emergent test-time scaling.
Existing large language model (LLM)-based embeddings typically adopt an encoder-only paradigm, treating LLMs as static feature extractors and overlooking their core generative strengths. We introduce GIRCSE (Generative Iterative Refinement for Contrastive Sentence Embeddings), a novel framework that leverages autoregressive generation to iteratively refine semantic representations. By producing sequences of soft tokens optimized under contrastive objective, GIRCSE captures latent concepts and implicit semantics that encoder-only methods often miss. To guide this process, we propose an Iterative Contrastive Refinement (ICR) objective that encourages each refinement step to yield better representations. Extensive experiments show that GIRCSE outperforms strong LLM-based embedding baselines on the MTEB benchmark and instruction-following tasks. Moreover, GIRCSE exhibits an emergent test-time scaling property: generating more tokens at inference steadily improves embedding quality. Our results establish generative iterative refinement as a new paradigm for representation learning.