Language Model Re-rankers are Fooled by Lexical Similarities
This work reveals critical limitations in LM re-rankers for retrieval-augmented generation, which could impact developers relying on them for semantic accuracy, though it is incremental in analyzing existing methods.
The study found that language model re-rankers often fail to outperform simple BM25 baselines on datasets like DRUID, struggling with lexical dissimilarities, and identified weaknesses that highlight the need for more adversarial evaluation datasets.
Language model (LM) re-rankers are used to refine retrieval results for retrieval-augmented generation (RAG). They are more expensive than lexical matching methods like BM25 but assumed to better process semantic information and the relations between the query and the retrieved answers. To understand whether LM re-rankers always live up to this assumption, we evaluate 6 different LM re-rankers on the NQ, LitQA2 and DRUID datasets. Our results show that LM re-rankers struggle to outperform a simple BM25 baseline on DRUID. Leveraging a novel separation metric based on BM25 scores, we explain and identify re-ranker errors stemming from lexical dissimilarities. We also investigate different methods to improve LM re-ranker performance and find these methods mainly useful for NQ. Taken together, our work identifies and explains weaknesses of LM re-rankers and points to the need for more adversarial and realistic datasets for their evaluation.