Learning Rank Functionals: An Empirical Study
This work addresses ranking problems in applications like information retrieval and recommender systems, but it is incremental as it builds on existing learning-to-rank methods without introducing a new paradigm.
The paper investigates design aspects for learning-to-rank algorithms, focusing on data representation, rank functionals, and loss functions correlated with evaluation metrics, and evaluates these on answer ranking and web information retrieval tasks with unspecified performance numbers.
Ranking is a key aspect of many applications, such as information retrieval, question answering, ad placement and recommender systems. Learning to rank has the goal of estimating a ranking model automatically from training data. In practical settings, the task often reduces to estimating a rank functional of an object with respect to a query. In this paper, we investigate key issues in designing an effective learning to rank algorithm. These include data representation, the choice of rank functionals, the design of the loss function so that it is correlated with the rank metrics used in evaluation. For the loss function, we study three techniques: approximating the rank metric by a smooth function, decomposition of the loss into a weighted sum of element-wise losses and into a weighted sum of pairwise losses. We then present derivations of piecewise losses using the theory of high-order Markov chains and Markov random fields. In experiments, we evaluate these design aspects on two tasks: answer ranking in a Social Question Answering site, and Web Information Retrieval.