Deep Reinforcement Learning with Embedded LQR Controllers
This addresses a specific control issue in robotics or autonomous systems, presenting an incremental improvement over existing methods.
The paper tackles the problem of chattering in goal states for reinforcement learning in reaching tasks by combining RL with classical LQR control, showing that adding LQR control improves performance, with more profound effects when augmenting discrete action sets.
Reinforcement learning is a model-free optimal control method that optimizes a control policy through direct interaction with the environment. For reaching tasks that end in regulation, popular discrete-action methods are not well suited due to chattering in the goal state. We compare three different ways to solve this problem through combining reinforcement learning with classical LQR control. In particular, we introduce a method that integrates LQR control into the action set, allowing generalization and avoiding fixing the computed control in the replay memory if it is based on learned dynamics. We also embed LQR control into a continuous-action method. In all cases, we show that adding LQR control can improve performance, although the effect is more profound if it can be used to augment a discrete action set.