Off-policy reinforcement learning for $ H_\infty $ control design
This addresses the challenge of solving Hamilton-Jacobi-Isaacs equations for control design in practical systems where accurate models are unavailable, though it appears incremental as it builds on existing RL and neural network techniques.
The paper tackled the nonlinear H∞ control design problem for systems with unknown models by introducing an off-policy reinforcement learning method to learn solutions from real data instead of relying on mathematical models, and proved its convergence while testing it on an F16 aircraft and a rotational/translational actuator system.
The $H_\infty$ control design problem is considered for nonlinear systems with unknown internal system model. It is known that the nonlinear $ H_\infty $ control problem can be transformed into solving the so-called Hamilton-Jacobi-Isaacs (HJI) equation, which is a nonlinear partial differential equation that is generally impossible to be solved analytically. Even worse, model-based approaches cannot be used for approximately solving HJI equation, when the accurate system model is unavailable or costly to obtain in practice. To overcome these difficulties, an off-policy reinforcement leaning (RL) method is introduced to learn the solution of HJI equation from real system data instead of mathematical system model, and its convergence is proved. In the off-policy RL method, the system data can be generated with arbitrary policies rather than the evaluating policy, which is extremely important and promising for practical systems. For implementation purpose, a neural network (NN) based actor-critic structure is employed and a least-square NN weight update algorithm is derived based on the method of weighted residuals. Finally, the developed NN-based off-policy RL method is tested on a linear F16 aircraft plant, and further applied to a rotational/translational actuator system.