Noisy Self-Training with Synthetic Queries for Dense Retrieval
This addresses the data efficiency challenge for dense retrieval systems, offering an incremental improvement over existing methods.
The paper tackles the problem of costly annotated data for neural retrieval models by introducing a noisy self-training framework with synthetic queries, which improves performance on general-domain and out-of-domain benchmarks, achieving gains with as little as 30% of labelled data.
Although existing neural retrieval models reveal promising results when training data is abundant and the performance keeps improving as training data increases, collecting high-quality annotated data is prohibitively costly. To this end, we introduce a novel noisy self-training framework combined with synthetic queries, showing that neural retrievers can be improved in a self-evolution manner with no reliance on any external models. Experimental results show that our method improves consistently over existing methods on both general-domain (e.g., MS-MARCO) and out-of-domain (i.e., BEIR) retrieval benchmarks. Extra analysis on low-resource settings reveals that our method is data efficient and outperforms competitive baselines, with as little as 30% of labelled training data. Further extending the framework for reranker training demonstrates that the proposed method is general and yields additional gains on tasks of diverse domains.\footnote{Source code is available at \url{https://github.com/Fantabulous-J/Self-Training-DPR}}