MarlRank: Multi-agent Reinforced Learning to Rank
This addresses the ranking problem in information retrieval by incorporating document interactions, though it is incremental as it builds on existing reinforcement learning and ranking methods.
The paper tackles the problem of ranking models neglecting mutual information among documents by proposing MarlRank, a multi-agent reinforced ranking model that formulates ranking as a multi-agent MDP, resulting in significant performance gains over state-of-the-art baselines on LETOR benchmark datasets.
When estimating the relevancy between a query and a document, ranking models largely neglect the mutual information among documents. A common wisdom is that if two documents are similar in terms of the same query, they are more likely to have similar relevance score. To mitigate this problem, in this paper, we propose a multi-agent reinforced ranking model, named MarlRank. In particular, by considering each document as an agent, we formulate the ranking process as a multi-agent Markov Decision Process (MDP), where the mutual interactions among documents are incorporated in the ranking process. To compute the ranking list, each document predicts its relevance to a query considering not only its own query-document features but also its similar documents features and actions. By defining reward as a function of NDCG, we can optimize our model directly on the ranking performance measure. Our experimental results on two LETOR benchmark datasets show that our model has significant performance gains over the state-of-art baselines. We also find that the NDCG shows an overall increasing trend along with the step of interactions, which demonstrates that the mutual information among documents helps improve the ranking performance.