LLM-based Listwise Reranking under the Effect of Positional Bias
This addresses a specific problem in information retrieval for researchers and practitioners by mitigating positional bias in reranking, though it is incremental as it builds on existing debiasing methods.
The paper tackles positional bias in LLM-based listwise passage reranking, where passages at the end are less likely to be ranked highly, by proposing DebiasFirst, a method that integrates positional calibration and position-aware data augmentation during fine-tuning, resulting in enhanced effectiveness and robustness across diverse first-stage retrievers and reduced dependence of NDCG@10 performance on document position.
LLM-based listwise passage reranking has attracted attention for its effectiveness in ranking candidate passages. However, these models suffer from positional bias, where passages positioned towards the end of the input are less likely to be moved to top positions in the ranking. We hypothesize that there are two primary sources of positional bias: (1) architectural bias inherent in LLMs and (2) the imbalanced positioning of relevant documents. To address this, we propose DebiasFirst, a method that integrates positional calibration and position-aware data augmentation during fine-tuning. Positional calibration uses inverse propensity scoring to adjust for positional bias by re-weighting the contributions of different positions in the loss function when training. Position-aware augmentation augments training data to ensure that each passage appears equally across varied positions in the input list. This approach markedly enhances both effectiveness and robustness to the original ranking across diverse first-stage retrievers, reducing the dependence of NDCG@10 performance on the position of relevant documents. DebiasFirst also complements the inference-stage debiasing methods, offering a practical solution for mitigating positional bias in reranking.