Overview of the TREC 2021 deep learning track
This work addresses information retrieval challenges for researchers and practitioners, but it is incremental as it builds on previous years' efforts.
The TREC 2021 Deep Learning track tackled information retrieval tasks using refreshed and expanded MS MARCO datasets, finding that deep neural ranking models with large-scale pretraining continued to outperform traditional methods, though single-stage retrieval did not match multi-stage pipelines.
This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we refreshed both the document and the passage collections which also led to a nearly four times increase in the document collection size and nearly $16$ times increase in the size of the passage collection. Deep neural ranking models that employ large scale pretraininig continued to outperform traditional retrieval methods this year. We also found that single stage retrieval can achieve good performance on both tasks although they still do not perform at par with multistage retrieval pipelines. Finally, the increase in the collection size and the general data refresh raised some questions about completeness of NIST judgments and the quality of the training labels that were mapped to the new collections from the old ones which we discuss in this report.