DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
This work addresses the efficiency-generalization trade-off in dense retrieval for practical deployment, though it is incremental as it builds on existing contrastive learning and LLM augmentation methods.
The paper tackles the problem of dense retrievers being either too slow (large language models) or too data-hungry (smaller models) by introducing DRAMA, a framework that uses LLMs to generate diverse training data for smaller retrievers, resulting in strong performance across multiple tasks and languages.
Large language models (LLMs) have demonstrated strong effectiveness and robustness while fine-tuned as dense retrievers. However, their large parameter size brings significant inference time computational challenges, including high encoding costs for large-scale corpora and increased query latency, limiting their practical deployment. While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data. In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers. In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup. Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong performance across multiple tasks and languages. These highlight the potential of connecting the training of smaller retrievers with the growing advancements in LLMs, bridging the gap between efficiency and generalization.