Stop Overthinking: Unlocking Efficient Listwise Reranking with Minimal Reasoning
For practitioners deploying reasoning-enhanced rerankers in latency-sensitive applications, this work offers a practical solution to reduce computational cost without sacrificing ranking quality.
The paper identifies an overthinking phenomenon in listwise reranking with LLMs, where extended reasoning yields diminishing returns, and proposes a Length-Regularized Self-Distillation framework that reduces inference token consumption by 34%-37% while maintaining effectiveness on TREC Deep Learning and NeuCLIR benchmarks.
Listwise reranking utilizing Large Language Models (LLMs) has achieved state-of-the-art retrieval effectiveness. Recently, reasoning-enhanced models have further pushed these boundaries by employing Chain-of-Thought (CoT) to perform deep comparative analysis of candidate documents. However, this performance gain comes at a prohibitive computational cost, as models often generate thousands of reasoning tokens before producing a final ranking. In this work, we investigate the relationship between reasoning length and ranking quality, revealing an overthinking phenomenon where extended reasoning yields diminishing returns. To address this, we propose a Length-Regularized Self-Distillation framework. We synthesize a dataset by sampling diverse reasoning traces from a teacher model (Rank-K) and applying a Pareto-inspired filter to select traces that achieve high ranking performance with minimal token usage. By fine-tuning on these concise, high-quality rationales, the student model learns to internalize efficient reasoning patterns, effectively pruning redundant deliberation. Experiments on TREC Deep Learning and NeuCLIR benchmarks demonstrate that our method maintains the teacher's effectiveness while reducing inference token consumption by 34%-37% across different retrieval settings, offering a practical solution for deploying reasoning-enhanced rerankers in latency-sensitive applications.