Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning
This addresses safety and multi-objective optimization in RL for real-world applications, but it is incremental as it builds on existing constrained optimization methods.
The paper tackles the problem of managing gradient conflicts and safety constraints in multi-objective reinforcement learning by proposing CoMOGA, which formulates it as a constrained optimization problem and achieves constraint satisfaction across all tasks.
In many real-world applications, a reinforcement learning (RL) agent should consider multiple objectives and adhere to safety guidelines. To address these considerations, we propose a constrained multi-objective RL algorithm named Constrained Multi-Objective Gradient Aggregator (CoMOGA). In the field of multi-objective optimization, managing conflicts between the gradients of the multiple objectives is crucial to prevent policies from converging to local optima. It is also essential to efficiently handle safety constraints for stable training and constraint satisfaction. We address these challenges straightforwardly by treating the maximization of multiple objectives as a constrained optimization problem (COP), where the constraints are defined to improve the original objectives. Existing safety constraints are then integrated into the COP, and the policy is updated using a linear approximation, which ensures the avoidance of gradient conflicts. Despite its simplicity, CoMOGA guarantees optimal convergence in tabular settings. Through various experiments, we have confirmed that preventing gradient conflicts is critical, and the proposed method achieves constraint satisfaction across all tasks.