Monte Carlo Rollout Policy for Recommendation Systems with Dynamic User Behavior

This work addresses the challenge of optimizing recommendation systems for dynamic user behavior, which is a problem for e-commerce platforms and content providers, with an incremental contribution to policy design.

This paper models online recommendation systems as a hidden Markov multi-state restless multi-armed bandit problem. It introduces a Monte Carlo rollout policy and numerically demonstrates its superior performance over a myopic policy when transition dynamics lack specific structure. However, the myopic policy outperforms the Monte Carlo rollout policy when structure is imposed on the transition dynamics.