Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning
This work addresses the problem of misalignment between short-term proxies and long-term goals in recommendation systems for users and platforms, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the challenge of optimizing long-term user utility in session-based recommendation systems, where delayed and confounded signals make it difficult. It introduces the Short Horizon Policy Improvement (SHPI) algorithm, which outperforms state-of-the-art methods like matrix factorization, bandits, and RL baselines on four recommendation tasks.
We study session-based recommendation scenarios where we want to recommend items to users during sequential interactions to improve their long-term utility. Optimizing a long-term metric is challenging because the learning signal (whether the recommendations achieved their desired goals) is delayed and confounded by other user interactions with the system. Targeting immediately measurable proxies such as clicks can lead to suboptimal recommendations due to misalignment with the long-term metric. We develop a new reinforcement learning algorithm called Short Horizon Policy Improvement (SHPI) that approximates policy-induced drift in user behavior across sessions. SHPI is a straightforward modification of episodic RL algorithms for session-based recommendation, that additionally gives an appropriate termination bonus in each session. Empirical results on four recommendation tasks show that SHPI can outperform state-of-the-art recommendation techniques like matrix factorization with offline proxy signals, bandits with myopic online proxies, and RL baselines with limited amounts of user interaction.