Multi-Action Restless Bandits with Weakly Coupled Constraints: Simultaneous Learning and Control
This work addresses the challenge of online decision-making in constrained bandit systems for applications like resource allocation, offering a novel approach but with incremental improvements over existing offline methods.
The paper tackles the problem of online multi-action restless bandits with weakly coupled constraints, where transition matrices and reward functions are unknown, by proposing a scheme for simultaneous learning and control. It proves convergence in time and magnitude dimensions, with exponential convergence in magnitude leading to exponentially diminishing performance deviation from offline optimality.
We study a system with finitely many groups of multi-action bandit processes, each of which is a Markov decision process (MDP) with finite state and action spaces and potentially different transition matrices when taking different actions. The bandit processes of the same group share the same state and action spaces and, given the same action that is taken, the same transition matrix. All the bandit processes across various groups are subject to multiple weakly coupled constraints over their state and action variables. Unlike the past studies that focused on the offline case, we consider the online case without assuming full knowledge of transition matrices and reward functions a priori and propose an effective scheme that enables simultaneous learning and control. We prove the convergence of the relevant processes in both the timeline and the number of the bandit processes, referred to as the convergence in the time and the magnitude dimensions. Moreover, we prove that the relevant processes converge exponentially fast in the magnitude dimension, leading to exponentially diminishing performance deviation between the proposed online algorithms and offline optimality.