Anytime-Competitive Reinforcement Learning with Policy Prior
This addresses the issue of high episodic costs in constrained reinforcement learning for applications like sustainable computing, though it is incremental as it builds on existing CMDP frameworks.
The paper tackles the problem of ensuring bounded cost in every episode of reinforcement learning, rather than just on average, by introducing Anytime-Competitive Markov Decision Processes (A-CMDP) and proposing the Anytime-Competitive Reinforcement Learning (ACRL) algorithm, which provably guarantees these constraints and asymptotically matches optimal reward under them, with experiments on carbon-intelligent computing verifying performance.
This paper studies the problem of Anytime-Competitive Markov Decision Process (A-CMDP). Existing works on Constrained Markov Decision Processes (CMDPs) aim to optimize the expected reward while constraining the expected cost over random dynamics, but the cost in a specific episode can still be unsatisfactorily high. In contrast, the goal of A-CMDP is to optimize the expected reward while guaranteeing a bounded cost in each round of any episode against a policy prior. We propose a new algorithm, called Anytime-Competitive Reinforcement Learning (ACRL), which provably guarantees the anytime cost constraints. The regret analysis shows the policy asymptotically matches the optimal reward achievable under the anytime competitive constraints. Experiments on the application of carbon-intelligent computing verify the reward performance and cost constraint guarantee of ACRL.