Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
This work addresses the challenge of learning effective recommendation policies for session-based recommendation systems in a purely offline setting, which is crucial for businesses to avoid costly online interactions and biased user models.
This paper tackles the problem of learning recommendation policies from offline historical interaction logs without online interaction or user-behavior models. The proposed BCD4Rec method, which combines batch and distributional reinforcement learning, significantly improves upon the behavior policy and other baselines in terms of Click Through Rates and Buy Rates.
Most of the existing deep reinforcement learning (RL) approaches for session-based recommendations either rely on costly online interactions with real users, or rely on potentially biased rule-based or data-driven user-behavior models for learning. In this work, we instead focus on learning recommendation policies in the pure batch or offline setting, i.e. learning policies solely from offline historical interaction logs or batch data generated from an unknown and sub-optimal behavior policy, without further access to data from the real-world or user-behavior models. We propose BCD4Rec: Batch-Constrained Distributional RL for Session-based Recommendations. BCD4Rec builds upon the recent advances in batch (offline) RL and distributional RL to learn from offline logs while dealing with the intrinsically stochastic nature of rewards from the users due to varied latent interest preferences (environments). We demonstrate that BCD4Rec significantly improves upon the behavior policy as well as strong RL and non-RL baselines in the batch setting in terms of standard performance metrics like Click Through Rates or Buy Rates. Other useful properties of BCD4Rec include: i. recommending items from the correct latent categories indicating better value estimates despite large action space (of the order of number of items), and ii. overcoming popularity bias in clicked or bought items typically present in the offline logs.