Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval
This addresses the limitation of encoder-only retrievers in handling reasoning-intensive queries for information retrieval tasks, though it is incremental as it builds on existing LLM embedding techniques.
The paper tackled the problem of complex queries requiring reasoning for dense retrieval by proposing RITE, which integrates logical reasoning into text embeddings using LLMs, and it significantly enhanced zero-shot retrieval performance on the BRIGHT benchmark.
Transformer-based models such as BERT and E5 have significantly advanced text embedding by capturing rich contextual representations. However, many complex real-world queries require sophisticated reasoning to retrieve relevant documents beyond surface-level lexical matching, where encoder-only retrievers often fall short. Decoder-only large language models (LLMs), known for their strong reasoning capabilities, offer a promising alternative. Despite this potential, existing LLM-based embedding methods primarily focus on contextual representation and do not fully exploit the reasoning strength of LLMs. To bridge this gap, we propose Reasoning-Infused Text Embedding (RITE), a simple but effective approach that integrates logical reasoning into the text embedding process using generative LLMs. RITE builds upon existing language model embedding techniques by generating intermediate reasoning texts in the token space before computing embeddings, thereby enriching representations with inferential depth. Experimental results on BRIGHT, a reasoning-intensive retrieval benchmark, demonstrate that RITE significantly enhances zero-shot retrieval performance across diverse domains, underscoring the effectiveness of incorporating reasoning into the embedding process.