Density Constrained Reinforcement Learning
This work addresses the problem of efficiently encoding constraints in reinforcement learning for applications like safety and resource management, offering a novel approach that avoids manual cost function tuning.
The paper tackles constrained reinforcement learning by imposing constraints on state density functions instead of value functions, leading to an algorithm that guarantees constraint satisfaction and converges to near-optimal solutions with bounded error, achieving superior performance in experiments on tasks like Safety-Gym.
We study constrained reinforcement learning (CRL) from a novel perspective by setting constraints directly on state density functions, rather than the value functions considered by previous works. State density has a clear physical and mathematical interpretation, and is able to express a wide variety of constraints such as resource limits and safety requirements. Density constraints can also avoid the time-consuming process of designing and tuning cost functions required by value function-based constraints to encode system specifications. We leverage the duality between density functions and Q functions to develop an effective algorithm to solve the density constrained RL problem optimally and the constrains are guaranteed to be satisfied. We prove that the proposed algorithm converges to a near-optimal solution with a bounded error even when the policy update is imperfect. We use a set of comprehensive experiments to demonstrate the advantages of our approach over state-of-the-art CRL methods, with a wide range of density constrained tasks as well as standard CRL benchmarks such as Safety-Gym.