A Reduction Algorithm for Markovian Contextual Linear Bandits
This work addresses bandit problems with correlated contexts, which is incremental but useful for applications like recommendation systems with temporal availability patterns.
The paper tackles the problem of linear contextual bandits with Markovian contexts by extending the 'contexts are cheap' reduction to handle temporally correlated action sets, achieving a regret bound that matches the underlying linear bandit oracle with minimal dependence on mixing time.
Recent work shows that when contexts are drawn i.i.d., linear contextual bandits can be reduced to single-context linear bandits. This ``contexts are cheap" perspective is highly advantageous, as it allows for sharper finite-time analyses and leverages mature techniques from the linear bandit literature, such as those for misspecification and adversarial corruption. Motivated by applications with temporally correlated availability, we extend this perspective to Markovian contextual linear bandits, where the action set evolves via an exogenous Markov chain. Our main contribution is a reduction that applies under uniform geometric ergodicity. We construct a stationary surrogate action set to solve the problem using a standard linear bandit oracle, employing a delayed-update scheme to control the bias induced by the nonstationary conditional context distributions. We further provide a phased algorithm for unknown transition distributions that learns the surrogate mapping online. In both settings, we obtain a high-probability worst-case regret bound matching that of the underlying linear bandit oracle, with only lower-order dependence on the mixing time.