Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders
For practitioners using LLMs as text encoders, EPIC offers a more efficient and effective way to leverage in-context learning for embedding generation.
EPIC replaces discrete text demonstrations with continuous embeddings for in-context learning, reducing token overhead while improving LLM-based text embeddings. It achieves state-of-the-art results on the MTEB benchmark, surpassing models trained on public retrieval data.
Large language models (LLMs) have been widely explored for embedding generation. While recent studies show that in-context learning (ICL) effectively enhances the representational capability of LLMs by prepending a few task-related demonstrations, it causes substantial token overhead due to the increased sequence length. In this work, we propose EPIC, a novel embedding-based in-context prompt training strategy that leverages ICL to generate high-quality embeddings while reducing computational burden during both training and inference. This approach replaces discrete text demonstrations with their corresponding continuous embeddings, which not only encourages the LLM to align semantically-related text pairs during contrastive learning, but also requires the model to interpret demonstration embeddings as part of the in-context prompt. Consequently, EPIC-trained models achieve excellent embedding performance both with or without in-context prompts at inference time. Comprehensive experiments demonstrate that our method establishes new state-of-the-art results on the MTEB benchmark, surpassing frontier models trained solely on publicly available retrieval data. Extensive ablation studies further validate the effectiveness and necessity of our mechanism.