A framework for reinforcement learning with autocorrelated actions
This addresses a practical issue in robotics by reducing shaking, though it appears incremental as it modifies existing RL approaches rather than introducing a new paradigm.
The paper tackles the problem of unwanted shaking in robots caused by white noise in reinforcement learning policies by introducing a framework with autocorrelated actions, which outperformed three baseline methods in three out of four simulated control tasks.
The subject of this paper is reinforcement learning. Policies are considered here that produce actions based on states and random elements autocorrelated in subsequent time instants. Consequently, an agent learns from experiments that are distributed over time and potentially give better clues to policy improvement. Also, physical implementation of such policies, e.g. in robotics, is less problematic, as it avoids making robots shake. This is in opposition to most RL algorithms which add white noise to control causing unwanted shaking of the robots. An algorithm is introduced here that approximately optimizes the aforementioned policy. Its efficiency is verified for four simulated learning control problems (Ant, HalfCheetah, Hopper, and Walker2D) against three other methods (PPO, SAC, ACER). The algorithm outperforms others in three of these problems.