Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval
This work addresses the challenge of domain adaptation for dense retrieval models, which is crucial for applications in information retrieval and search systems, though it appears incremental as it builds on existing distillation and reranking techniques.
The paper tackles the problem of limited domain transfer ability in neural dense retrieval models by introducing ABEL, an unsupervised method that alternates distillation between a dense retriever and a reranker to enhance zero-shot passage retrieval. The results show that ABEL outperforms leading supervised and unsupervised retrievers on the BEIR benchmark and achieves state-of-the-art performance when fine-tuned or integrated with supervised models.
Neural 'dense' retrieval models are state of the art for many datasets, however these models often exhibit limited domain transfer ability. Existing approaches to adaptation are unwieldy, such as requiring explicit supervision, complex model architectures, or massive external models. We present $\texttt{ABEL}$, a simple but effective unsupervised method to enhance passage retrieval in zero-shot settings. Our technique follows a straightforward loop: a dense retriever learns from supervision signals provided by a reranker, and subsequently, the reranker is updated based on feedback from the improved retriever. By iterating this loop, the two components mutually enhance one another's performance. Experimental results demonstrate that our unsupervised $\texttt{ABEL}$ model outperforms both leading supervised and unsupervised retrievers on the BEIR benchmark. Meanwhile, it exhibits strong adaptation abilities to tasks and domains that were unseen during training. By either fine-tuning $\texttt{ABEL}$ on labelled data or integrating it with existing supervised dense retrievers, we achieve state-of-the-art results.\footnote{Source code is available at \url{https://github.com/Fantabulous-J/BootSwitch}.}