Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback
This work addresses a key bottleneck in recommendation systems by enabling offline optimization for long-term rewards, which is incremental but practical for domains like ranking and recommendation.
The paper tackles the problem of optimizing ranking policies for long-term user rewards without online interactions, proposing an off-policy value ranking algorithm that integrates future rewards and ranking metrics in an EM framework, achieving improved sample efficiency and effectiveness in experiments.
Probabilistic learning to rank (LTR) has been the dominating approach for optimizing the ranking metric, but cannot maximize long-term rewards. Reinforcement learning models have been proposed to maximize user long-term rewards by formulating the recommendation as a sequential decision-making problem, but could only achieve inferior accuracy compared to LTR counterparts, primarily due to the lack of online interactions and the characteristics of ranking. In this paper, we propose a new off-policy value ranking (VR) algorithm that can simultaneously maximize user long-term rewards and optimize the ranking metric offline for improved sample efficiency in a unified Expectation-Maximization (EM) framework. We theoretically and empirically show that the EM process guides the leaned policy to enjoy the benefit of integration of the future reward and ranking metric, and learn without any online interactions. Extensive offline and online experiments demonstrate the effectiveness of our methods.