PiRank: Scalable Learning To Rank via Differentiable Sorting
This work is significant for researchers and practitioners in learning-to-rank, as it provides a scalable and more accurate method for optimizing ranking models by bridging the gap between loss functions and performance metrics.
This paper addresses the challenge of optimizing non-differentiable ranking metrics in machine learning by introducing PiRank, a new class of differentiable surrogates. PiRank utilizes a continuous, temperature-controlled relaxation of the sorting operator and scales efficiently to large list sizes through a divide-and-conquer extension, leading to significant performance improvements on internet-scale learning-to-rank benchmarks.
A key challenge with machine learning approaches for ranking is the gap between the performance metrics of interest and the surrogate loss functions that can be optimized with gradient-based methods. This gap arises because ranking metrics typically involve a sorting operation which is not differentiable w.r.t. the model parameters. Prior works have proposed surrogates that are loosely related to ranking metrics or simple smoothed versions thereof, and often fail to scale to real-world applications. We propose PiRank, a new class of differentiable surrogates for ranking, which employ a continuous, temperature-controlled relaxation to the sorting operator based on NeuralSort [1]. We show that PiRank exactly recovers the desired metrics in the limit of zero temperature and further propose a divide and-conquer extension that scales favorably to large list sizes, both in theory and practice. Empirically, we demonstrate the role of larger list sizes during training and show that PiRank significantly improves over comparable approaches on publicly available internet-scale learning-to-rank benchmarks.