Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology
This work addresses the problem of long-term recommendation optimization for users in platforms with slates, offering a practical incremental improvement over existing myopic methods.
The paper tackles the challenge of optimizing long-term user engagement in slate-based recommender systems by introducing SLATEQ, a decomposition method that makes reinforcement learning tractable for slates, and demonstrates its scalability in live experiments on YouTube.
Most practical recommender systems focus on estimating immediate user engagement without considering the long-term effects of recommendations on user behavior. Reinforcement learning (RL) methods offer the potential to optimize recommendations for long-term user engagement. However, since users are often presented with slates of multiple items - which may have interacting effects on user choice - methods are required to deal with the combinatorics of the RL action space. In this work, we address the challenge of making slate-based recommendations to optimize long-term value using RL. Our contributions are three-fold. (i) We develop SLATEQ, a decomposition of value-based temporal-difference and Q-learning that renders RL tractable with slates. Under mild assumptions on user choice behavior, we show that the long-term value (LTV) of a slate can be decomposed into a tractable function of its component item-wise LTVs. (ii) We outline a methodology that leverages existing myopic learning-based recommenders to quickly develop a recommender that handles LTV. (iii) We demonstrate our methods in simulation, and validate the scalability of decomposed TD-learning using SLATEQ in live experiments on YouTube.