MDDL: A Framework for Reinforcement Learning-based Position Allocation in Multi-Channel Feed
This addresses the challenge of effectively utilizing mixed data distributions for RL training in industrial recommendation systems, though it appears incremental as it builds on existing RL methods for a specific bottleneck.
The paper tackles the problem of training reinforcement learning models for position allocation in multi-channel feeds using both strategy and random data, which have different distributions, by proposing the MDDL framework that incorporates imitation learning to mitigate overestimation and maximizes RL signals, resulting in superior performance in a real-world system deployed on Meituan serving over 300 million users.
Nowadays, the mainstream approach in position allocation system is to utilize a reinforcement learning model to allocate appropriate locations for items in various channels and then mix them into the feed. There are two types of data employed to train reinforcement learning (RL) model for position allocation, named strategy data and random data. Strategy data is collected from the current online model, it suffers from an imbalanced distribution of state-action pairs, resulting in severe overestimation problems during training. On the other hand, random data offers a more uniform distribution of state-action pairs, but is challenging to obtain in industrial scenarios as it could negatively impact platform revenue and user experience due to random exploration. As the two types of data have different distributions, designing an effective strategy to leverage both types of data to enhance the efficacy of the RL model training has become a highly challenging problem. In this study, we propose a framework named Multi-Distribution Data Learning (MDDL) to address the challenge of effectively utilizing both strategy and random data for training RL models on mixed multi-distribution data. Specifically, MDDL incorporates a novel imitation learning signal to mitigate overestimation problems in strategy data and maximizes the RL signal for random data to facilitate effective learning. In our experiments, we evaluated the proposed MDDL framework in a real-world position allocation system and demonstrated its superior performance compared to the previous baseline. MDDL has been fully deployed on the Meituan food delivery platform and currently serves over 300 million users.