Is Learning to Rank Worth It? A Statistical Analysis of Learning to Rank Methods
This challenges the core assumption in the L2R field, suggesting incremental or limited practical gains for researchers and practitioners in information retrieval.
The study investigated whether sophisticated Learning to Rank (L2R) algorithms significantly outperform simpler information retrieval methods, finding that many L2R algorithms do not produce statistically significant differences compared to the best individual features on benchmark datasets, with most baselines statistically tied.
The Learning to Rank (L2R) research field has experienced a fast paced growth over the last few years, with a wide variety of benchmark datasets and baselines available for experimentation. We here investigate the main assumption behind this field, which is that, the use of sophisticated L2R algorithms and models, produce significant gains over more traditional and simple information retrieval approaches. Our experimental results surprisingly indicate that many L2R algorithms, when put up against the best individual features of each dataset, may not produce statistically significant differences, even if the absolute gains may seem large. We also find that most of the reported baselines are statistically tied, with no clear winner.