Extended Radial Basis Function Controller for Reinforcement Learning
This work addresses the challenge of integrating control theory into reinforcement learning for improved stability, but it appears incremental as it builds on existing methods like PILCO and DDPG.
The paper tackles the problem of incorporating prior model knowledge into reinforcement learning by proposing a hybrid controller that dynamically interpolates a model-based linear controller and a differentiable policy, ensuring stability near an operating point while maintaining universal approximation. Simulation experiments in OpenAI gym demonstrated stability and robustness, though no concrete performance numbers were provided.
There have been attempts in reinforcement learning to exploit a priori knowledge about the structure of the system. This paper proposes a hybrid reinforcement learning controller which dynamically interpolates a model-based linear controller and an arbitrary differentiable policy. The linear controller is designed based on local linearised model knowledge, and stabilises the system in a neighbourhood about an operating point. The coefficients of interpolation between the two controllers are determined by a scaled distance function measuring the distance between the current state and the operating point. The overall hybrid controller is proven to maintain the stability guarantee around the neighborhood of the operating point and still possess the universal function approximation property of the arbitrary non-linear policy. Learning has been done on both model-based (PILCO) and model-free (DDPG) frameworks. Simulation experiments performed in OpenAI gym demonstrate stability and robustness of the proposed hybrid controller. This paper thus introduces a principled method allowing for the direct importing of control methodology into reinforcement learning.