Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control
This work addresses robust control for practical systems like UAVs, offering a more efficient alternative to complex $H_\infty$ methods, though it is incremental as it builds on existing DPG and TD3 techniques.
The paper tackles the challenge of designing robust controllers for systems with model uncertainties and external disturbances by proposing a reinforcement learning algorithm called robust deterministic policy gradient (RDPG), which formulates $H_\infty$ control as a two-player game and is extended to RDDPG with deep neural networks; experimental results on a UAV show it outperforms other methods in robustness and enables precise real-time tracking under severe disturbances.
Practical control systems pose significant challenges in identifying optimal control policies due to uncertainties in the system model and external disturbances. While $H_\infty$ control techniques are commonly used to design robust controllers that mitigate the effects of disturbances, these methods often require complex and computationally intensive calculations. To address this issue, this paper proposes a reinforcement learning algorithm called robust deterministic policy gradient (RDPG), which formulates the $H_\infty$ control problem as a two-player zero-sum dynamic game. In this formulation, one player (the user) aims to minimize the cost, while the other player (the adversary) seeks to maximize it. We then employ deterministic policy gradient (DPG) and its deep reinforcement learning counterpart to train a robust control policy with effective disturbance attenuation. In particular, for practical implementation, we introduce an algorithm called robust deep deterministic policy gradient (RDDPG), which employs a deep neural network architecture and integrates techniques from the twin-delayed deep deterministic policy gradient (TD3) to enhance stability and learning efficiency. To evaluate the proposed algorithm, we implement it on an unmanned aerial vehicle (UAV) tasked with following a predefined path in a disturbance-prone environment. The experimental results demonstrate that the proposed method outperforms other control approaches in terms of robustness against disturbances, enabling precise real-time tracking of moving targets even under severe disturbance conditions.