GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings
This addresses the need for efficient, training-free sentence embeddings for NLP practitioners, offering incremental improvements over prior prompt-based approaches.
The paper tackles the problem of generating sentence embeddings without training by proposing GenEOL, which uses LLMs to create diverse sentence transformations and aggregates their embeddings, resulting in an average improvement of 2.85 points on the STS benchmark over existing methods.
Training-free embedding methods directly leverage pretrained large language models (LLMs) to embed text, bypassing the costly and complex procedure of contrastive learning. Previous training-free embedding methods have mainly focused on optimizing embedding prompts and have overlooked the benefits of utilizing the generative abilities of LLMs. We propose a novel method, GenEOL, which uses LLMs to generate diverse transformations of a sentence that preserve its meaning, and aggregates the resulting embeddings of these transformations to enhance the overall sentence embedding. GenEOL significantly outperforms the existing training-free embedding methods by an average of 2.85 points across several LLMs on the sentence semantic text similarity (STS) benchmark. GenEOL also achieves notable gains in clustering, reranking, and pair-classification tasks from the MTEB benchmark. Additionally, GenEOL stabilizes representation quality across LLM layers and remains robust to perturbations of embedding prompts.