Making Large Language Models Efficient Dense Retrievers
This work addresses the problem of computational inefficiency for researchers and practitioners using LLMs in retrieval tasks, offering an incremental improvement by optimizing existing methods for redundancy.
The paper tackles the computational inefficiency of large language models (LLMs) used as dense retrievers by analyzing layer redundancy, finding that MLP layers are highly prunable while attention layers are critical. It proposes EffiR, a framework that compresses MLP layers through coarse-to-fine strategies, achieving substantial reductions in model size and inference cost while preserving performance across BEIR datasets and LLM backbones.
Recent work has shown that directly fine-tuning large language models (LLMs) for dense retrieval yields strong performance, but their substantial parameter counts make them computationally inefficient. While prior studies have revealed significant layer redundancy in LLMs for generative tasks, it remains unclear whether similar redundancy exists when these models are adapted for retrieval tasks, which require encoding entire sequences into fixed representations rather than generating tokens iteratively. To this end, we conduct a comprehensive analysis of layer redundancy in LLM-based dense retrievers. We find that, in contrast to generative settings, MLP layers are substantially more prunable, while attention layers remain critical for semantic aggregation. Building on this insight, we propose EffiR, a framework for developing efficient retrievers that performs large-scale MLP compression through a coarse-to-fine strategy (coarse-grained depth reduction followed by fine-grained width reduction), combined with retrieval-specific fine-tuning. Across diverse BEIR datasets and LLM backbones, EffiR achieves substantial reductions in model size and inference cost while preserving the performance of full-size models.