Budgeted Reinforcement Learning in Continuous State Space
This work addresses safety-critical applications like autonomous driving by enabling budgeted reinforcement learning in continuous environments, representing a significant extension of existing methods.
The authors tackled the problem of solving Budgeted Markov Decision Processes (BMDPs) in continuous state spaces with unknown dynamics, extending prior work limited to finite spaces. They introduced a novel Budgeted Bellman Optimality operator and validated their approach on simulated spoken dialogue and autonomous driving applications.
A Budgeted Markov Decision Process (BMDP) is an extension of a Markov Decision Process to critical applications requiring safety constraints. It relies on a notion of risk implemented in the shape of a cost signal constrained to lie below an - adjustable - threshold. So far, BMDPs could only be solved in the case of finite state spaces with known dynamics. This work extends the state-of-the-art to continuous spaces environments and unknown dynamics. We show that the solution to a BMDP is a fixed point of a novel Budgeted Bellman Optimality operator. This observation allows us to introduce natural extensions of Deep Reinforcement Learning algorithms to address large-scale BMDPs. We validate our approach on two simulated applications: spoken dialogue and autonomous driving.