Taming the Adversary: Stable Minimax Deep Deterministic Policy Gradient via Fractional Objectives
This addresses the critical challenge of ensuring reliable performance for RL agents in continuous control tasks under adversarial conditions, representing an incremental improvement in robust RL methods.
The paper tackles the problem of unstable performance in reinforcement learning agents under unexpected disturbances and model uncertainties by proposing a minimax deep deterministic policy gradient (MMDDPG) framework with a fractional objective, achieving significantly improved robustness in MuJoCo environments.
Reinforcement learning (RL) has achieved remarkable success in a wide range of control and decision-making tasks. However, RL agents often exhibit unstable or degraded performance when deployed in environments subject to unexpected external disturbances and model uncertainties. Consequently, ensuring reliable performance under such conditions remains a critical challenge. In this paper, we propose minimax deep deterministic policy gradient (MMDDPG), a framework for learning disturbance-resilient policies in continuous control tasks. The training process is formulated as a minimax optimization problem between a user policy and an adversarial disturbance policy. In this problem, the user learns a robust policy that minimizes the objective function, while the adversary generates disturbances that maximize it. To stabilize this interaction, we introduce a fractional objective that balances task performance and disturbance magnitude. This objective prevents excessively aggressive disturbances and promotes robust learning. Experimental evaluations in MuJoCo environments demonstrate that the proposed MMDDPG achieves significantly improved robustness against both external force perturbations and model parameter variations.