SumRank: Aligning Summarization Models for Long-Document Listwise Reranking
This addresses efficiency and effectiveness issues in long-document ranking for information retrieval, though it is incremental as it builds on existing summarization and reranking methods.
The paper tackles the problem of applying large language models to rank long documents by proposing SumRank, a summarization model aligned with listwise reranking, which achieves state-of-the-art performance on five TREC Deep Learning datasets while improving efficiency.
Large Language Models (LLMs) have demonstrated superior performance in listwise passage reranking task. However, directly applying them to rank long-form documents introduces both effectiveness and efficiency issues due to the substantially increased context length. To address this challenge, we propose a pointwise summarization model SumRank, aligned with downstream listwise reranking, to compress long-form documents into concise rank-aligned summaries before the final listwise reranking stage. To obtain our summarization model SumRank, we introduce a three-stage training pipeline comprising cold-start Supervised Fine-Tuning (SFT), specialized RL data construction, and rank-driven alignment via Reinforcement Learning. This paradigm aligns the SumRank with downstream ranking objectives to preserve relevance signals. We conduct extensive experiments on five benchmark datasets from the TREC Deep Learning tracks (TREC DL 19-23). Results show that our lightweight SumRank model achieves state-of-the-art (SOTA) ranking performance while significantly improving efficiency by reducing both summarization overhead and reranking complexity.