Latency-Aware Contextual Bandit: Application to Cryo-EM Data Collection
This work addresses the challenge of action delays in sequential decision-making for applications like cryo-EM data collection, representing an incremental improvement over standard contextual bandits.
The authors tackled the problem of selecting actions with delays in contextual bandits by introducing a latency-aware framework, showing that their COAF algorithm achieves regret bounds matching existing results and performs well in movie recommendation and cryo-EM data experiments.
We introduce a latency-aware contextual bandit framework that generalizes the standard contextual bandit problem, where the learner adaptively selects arms and switches decision sets under action delays. In this setting, the learner observes the context and may select multiple arms from a decision set, with the total time determined by the selected subset. The problem can be framed as a special case of semi-Markov decision processes (SMDPs), where contexts and latencies are drawn from an unknown distribution. Leveraging the Bellman optimality equation, we design the contextual online arm filtering (COAF) algorithm, which balances exploration, exploitation, and action latency to minimize regret relative to the optimal average-reward policy. We analyze the algorithm and show that its regret upper bounds match established results in the contextual bandit literature. In numerical experiments on a movie recommendation dataset and cryogenic electron microscopy (cryo-EM) data, we demonstrate that our approach efficiently maximizes cumulative reward over time.