LLM-Oriented Retrieval Tuner
This addresses the problem of efficient retrieval-augmented generation for LLM users, though it is incremental as it builds on existing dense retrieval methods.
The paper tackles the challenge of integrating dense retrieval with large language models without tuning the LLM itself, proposing LMORT to decouple retrieval capacity and achieve competitive zero-shot retrieval performance on six BEIR datasets while preserving generation ability.
Dense Retrieval (DR) is now considered as a promising tool to enhance the memorization capacity of Large Language Models (LLM) such as GPT3 and GPT-4 by incorporating external memories. However, due to the paradigm discrepancy between text generation of LLM and DR, it is still an open challenge to integrate the retrieval and generation tasks in a shared LLM. In this paper, we propose an efficient LLM-Oriented Retrieval Tuner, namely LMORT, which decouples DR capacity from base LLM and non-invasively coordinates the optimally aligned and uniform layers of the LLM towards a unified DR space, achieving an efficient and effective DR without tuning the LLM itself. The extensive experiments on six BEIR datasets show that our approach could achieve competitive zero-shot retrieval performance compared to a range of strong DR models while maintaining the generation ability of LLM.