Balancing Constraints and Rewards with Meta-Gradient D4PG
This addresses the problem of constraint satisfaction in RL for real-world applications where violations are undesirable but not catastrophic, representing an incremental improvement over existing methods.
The paper tackles the problem of reinforcement learning agents needing to satisfy complex system constraints in real-world applications where constraint thresholds are often incorrectly set, presenting a soft-constrained RL approach that uses meta-gradients to balance expected return and constraint violations. The result shows this approach consistently outperforms baselines across four MuJoCo domains.
Deploying Reinforcement Learning (RL) agents to solve real-world applications often requires satisfying complex system constraints. Often the constraint thresholds are incorrectly set due to the complex nature of a system or the inability to verify the thresholds offline (e.g, no simulator or reasonable offline evaluation procedure exists). This results in solutions where a task cannot be solved without violating the constraints. However, in many real-world cases, constraint violations are undesirable yet they are not catastrophic, motivating the need for soft-constrained RL approaches. We present a soft-constrained RL approach that utilizes meta-gradients to find a good trade-off between expected return and minimizing constraint violations. We demonstrate the effectiveness of this approach by showing that it consistently outperforms the baselines across four different MuJoCo domains.