A Thorough Examination on Zero-shot Dense Retrieval
This work addresses the gap in understanding zero-shot retrieval for researchers and practitioners in information retrieval, though it is incremental as it reviews and analyzes existing models rather than proposing a new method.
The paper tackles the problem of dense retrieval models underperforming in zero-shot settings compared to sparse models by conducting the first comprehensive examination of zero-shot dense retrieval, identifying key factors like training data and dataset bias that affect performance.
Recent years have witnessed the significant advance in dense retrieval (DR) based on powerful pre-trained language models (PLM). DR models have achieved excellent performance in several benchmark datasets, while they are shown to be not as competitive as traditional sparse retrieval models (e.g., BM25) in a zero-shot retrieval setting. However, in the related literature, there still lacks a detailed and comprehensive study on zero-shot retrieval. In this paper, we present the first thorough examination of the zero-shot capability of DR models. We aim to identify the key factors and analyze how they affect zero-shot retrieval performance. In particular, we discuss the effect of several key factors related to source training set, analyze the potential bias from the target dataset, and review and compare existing zero-shot DR models. Our findings provide important evidence to better understand and develop zero-shot DR models.