Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay
This work addresses the problem of improving long-term user engagement in recommender systems for online platforms, representing an incremental advance by combining existing techniques in a novel way for a known bottleneck.
The paper tackles the challenge of optimizing recommender systems for long-term user satisfaction by addressing the trade-off between delayed rewards and short-term proxies, formalizing it as a multi-armed bandit problem with delayed rewards. They develop a predictive model using a Bayesian filter and a bandit algorithm, applying it to podcast recommendations and achieving substantially better performance compared to baseline methods.
Recommender systems are a ubiquitous feature of online platforms. Increasingly, they are explicitly tasked with increasing users' long-term satisfaction. In this context, we study a content exploration task, which we formalize as a multi-armed bandit problem with delayed rewards. We observe that there is an apparent trade-off in choosing the learning signal: Waiting for the full reward to become available might take several weeks, hurting the rate at which learning happens, whereas measuring short-term proxy rewards reflects the actual long-term goal only imperfectly. We address this challenge in two steps. First, we develop a predictive model of delayed rewards that incorporates all information obtained to date. Full observations as well as partial (short or medium-term) outcomes are combined through a Bayesian filter to obtain a probabilistic belief. Second, we devise a bandit algorithm that takes advantage of this new predictive model. The algorithm quickly learns to identify content aligned with long-term success by carefully balancing exploration and exploitation. We apply our approach to a podcast recommendation problem, where we seek to identify shows that users engage with repeatedly over two months. We empirically validate that our approach results in substantially better performance compared to approaches that either optimize for short-term proxies, or wait for the long-term outcome to be fully realized.