Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment
This work addresses the need for more effective and versatile retrievers in information retrieval systems, though it is incremental as it builds on existing research by providing a comprehensive empirical assessment of LLMs.
The study tackled the problem of limited generalization and in-domain accuracy in dense retrieval by empirically assessing large language models (LLMs) as backbone encoders, finding that larger models and extensive pretraining consistently improve in-domain accuracy, data efficiency, and zero-shot generalization across various retrieval tasks.
Pretrained language models like BERT and T5 serve as crucial backbone encoders for dense retrieval. However, these models often exhibit limited generalization capabilities and face challenges in improving in domain accuracy. Recent research has explored using large language models (LLMs) as retrievers, achieving SOTA performance across various tasks. Despite these advancements, the specific benefits of LLMs over traditional retrievers and the impact of different LLM configurations, such as parameter sizes, pretraining duration, and alignment processes on retrieval tasks remain unclear. In this work, we conduct a comprehensive empirical study on a wide range of retrieval tasks, including in domain accuracy, data efficiency, zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. We evaluate over 15 different backbone LLMs and non LLMs. Our findings reveal that larger models and extensive pretraining consistently enhance in domain accuracy and data efficiency. Additionally, larger models demonstrate significant potential in zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. These results underscore the advantages of LLMs as versatile and effective backbone encoders in dense retrieval, providing valuable insights for future research and development in this field.