Generator and Critic: A Deep Reinforcement Learning Approach for Slate Re-ranking in E-commerce
This work addresses the challenge of optimizing item rankings for user satisfaction in e-commerce, though it is incremental as it builds on existing reinforcement learning methods for slate re-ranking.
The paper tackles the slate re-ranking problem in e-commerce by proposing a Generator and Critic approach using deep reinforcement learning, where a Full Slate Critic model evaluates slates and a PPO-Exploration algorithm generates rankings, resulting in improvements of 4% GMV and 5% number of orders in live experiments.
The slate re-ranking problem considers the mutual influences between items to improve user satisfaction in e-commerce, compared with the point-wise ranking. Previous works either directly rank items by an end to end model, or rank items by a score function that trades-off the point-wise score and the diversity between items. However, there are two main existing challenges that are not well studied: (1) the evaluation of the slate is hard due to the complex mutual influences between items of one slate; (2) even given the optimal evaluation, searching the optimal slate is challenging as the action space is exponentially large. In this paper, we present a novel Generator and Critic slate re-ranking approach, where the Critic evaluates the slate and the Generator ranks the items by the reinforcement learning approach. We propose a Full Slate Critic (FSC) model that considers the real impressed items and avoids the impressed bias of existing models. For the Generator, to tackle the problem of large action space, we propose a new exploration reinforcement learning algorithm, called PPO-Exploration. Experimental results show that the FSC model significantly outperforms the state of the art slate evaluation methods, and the PPO-Exploration algorithm outperforms the existing reinforcement learning methods substantially. The Generator and Critic approach improves both the slate efficiency(4% gmv and 5% number of orders) and diversity in live experiments on one of the largest e-commerce websites in the world.