Understanding the Behaviors of BERT in Ranking
This work addresses ranking challenges for information retrieval researchers, but it is incremental as it applies an existing method to new data and tasks.
The paper investigates BERT's performance in ranking tasks, showing it is highly effective for question-answering passage ranking on MS MARCO but has gaps for ad hoc document ranking on TREC, with analyses revealing attention patterns and semantic match preferences.
This paper studies the performances and behaviors of BERT in ranking tasks. We explore several different ways to leverage the pre-trained BERT and fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the strong effectiveness of BERT in question-answering focused passage ranking tasks, as well as the fact that BERT is a strong interaction-based seq2seq matching model. Experimental results on TREC show the gaps between the BERT pre-trained on surrounding contexts and the needs of ad hoc document ranking. Analyses illustrate how BERT allocates its attentions between query-document tokens in its Transformer layers, how it prefers semantic matches between paraphrase tokens, and how that differs with the soft match patterns learned by a click-trained neural ranker.