Learning and Optimization with Seasonal Patterns
This work addresses the challenge of time-varying rewards in decision-making for business applications, representing an incremental improvement over stationary bandit models.
The paper tackles the problem of non-stationary multi-armed bandits with periodic mean rewards, where unknown periods can vary across arms and scale polynomially with the horizon, by proposing a two-stage policy that learns these periods and achieves a regret upper bound of ̃O(√(T∑_k T_k)), which is near-optimal up to a factor of √K.
A standard assumption adopted in the multi-armed bandit (MAB) framework is that the mean rewards are constant over time. This assumption can be restrictive in the business world as decision-makers often face an evolving environment where the mean rewards are time-varying. In this paper, we consider a non-stationary MAB model with $K$ arms whose mean rewards vary over time in a periodic manner. The unknown periods can be different across arms and scale with the length of the horizon $T$ polynomially. We propose a two-stage policy that combines the Fourier analysis with a confidence-bound-based learning procedure to learn the periods and minimize the regret. In stage one, the policy correctly estimates the periods of all arms with high probability. In stage two, the policy explores the periodic mean rewards of arms using the periods estimated in stage one and exploits the optimal arm in the long run. We show that our learning policy incurs a regret upper bound $\tilde{O}(\sqrt{T\sum_{k=1}^K T_k})$ where $T_k$ is the period of arm $k$. Moreover, we establish a general lower bound $Ω(\sqrt{T\max_{k}\{ T_k\}})$ for any policy. Therefore, our policy is near-optimal up to a factor of $\sqrt{K}$.