Noise tolerance of learning to rank under class-conditional label noise
This addresses label noise issues in ranking tasks like web search, offering a theoretical and practical solution for practitioners, though it is incremental as it builds on existing noise-tolerant concepts.
The paper tackles the problem of label noise in learning-to-rank models, showing that a class of noise-tolerant losses enables consistent empirical risk minimization under class-conditional label noise, with experimental support for practical implications.
Often, the data used to train ranking models is subject to label noise. For example, in web-search, labels created from clickstream data are noisy due to issues such as insufficient information in item descriptions on the SERP, query reformulation by the user, and erratic or unexpected user behavior. In practice, it is difficult to handle label noise without making strong assumptions about the label generation process. As a result, practitioners typically train their learning-to-rank (LtR) models directly on this noisy data without additional consideration of the label noise. Surprisingly, we often see strong performance from LtR models trained in this way. In this work, we describe a class of noise-tolerant LtR losses for which empirical risk minimization is a consistent procedure, even in the context of class-conditional label noise. We also develop noise-tolerant analogs of commonly used loss functions. The practical implications of our theoretical findings are further supported by experimental results.