Drowning in Documents: Consequences of Scaling Reranker Inference
This addresses a critical issue for information retrieval practitioners by revealing limitations in scaling reranker inference, though it is incremental as it builds on existing assumptions.
The paper tackled the problem of reranker performance in full retrieval scenarios, finding that while rerankers initially improve with more documents, their effectiveness declines and can degrade quality beyond a certain limit, with specific degradation observed at thresholds like 100 documents.
Rerankers, typically cross-encoders, are computationally intensive but are frequently used because they are widely assumed to outperform cheaper initial IR systems. We challenge this assumption by measuring reranker performance for full retrieval, not just re-scoring first-stage retrieval. To provide a more robust evaluation, we prioritize strong first-stage retrieval using modern dense embeddings and test rerankers on a variety of carefully chosen, challenging tasks, including internally curated datasets to avoid contamination, and out-of-domain ones. Our empirical results reveal a surprising trend: the best existing rerankers provide initial improvements when scoring progressively more documents, but their effectiveness gradually declines and can even degrade quality beyond a certain limit. We hope that our findings will spur future research to improve reranking.