Learning to Rank from Relevance Judgments Distributions
This work addresses the limitation of single-label annotations in ranking systems, offering a method to leverage richer relevance data for improved model training, though it is incremental as it builds on existing LETOR frameworks.
The paper tackles the problem of training Learning to Rank models using relevance judgments distributions instead of single labels, proposing five new probabilistic loss functions and showing that this approach can boost performance and outperform strong baselines like LambdaMART on several test collections.
Learning to Rank (LETOR) algorithms are usually trained on annotated corpora where a single relevance label is assigned to each available document-topic pair. Within the Cranfield framework, relevance labels result from merging either multiple expertly curated or crowdsourced human assessments. In this paper, we explore how to train LETOR models with relevance judgments distributions (either real or synthetically generated) assigned to document-topic pairs instead of single-valued relevance labels. We propose five new probabilistic loss functions to deal with the higher expressive power provided by relevance judgments distributions and show how they can be applied both to neural and GBM architectures. Moreover, we show how training a LETOR model on a sampled version of the relevance judgments from certain probability distributions can improve its performance when relying either on traditional or probabilistic loss functions. Finally, we validate our hypothesis on real-world crowdsourced relevance judgments distributions. Overall, we observe that relying on relevance judgments distributions to train different LETOR models can boost their performance and even outperform strong baselines such as LambdaMART on several test collections.