Smooth Exploration for Robotic Reinforcement Learning
This work addresses the issue of shaky behavior in real robots during RL training, which can cause poor exploration or damage, making it significant for robotics applications but incremental in method adaptation.
The authors tackled the problem of jerky motion patterns in robotic reinforcement learning by adapting state-dependent exploration to deep RL algorithms, resulting in a new method called generalized state-dependent exploration (gSDE) that allows training directly on real robots without performance loss.
Reinforcement learning (RL) enables robots to learn skills from interactions with the real world. In practice, the unstructured step-based exploration used in Deep RL -- often very successful in simulation -- leads to jerky motion patterns on real robots. Consequences of the resulting shaky behavior are poor exploration, or even damage to the robot. We address these issues by adapting state-dependent exploration (SDE) to current Deep RL algorithms. To enable this adaptation, we propose two extensions to the original SDE, using more general features and re-sampling the noise periodically, which leads to a new exploration method generalized state-dependent exploration (gSDE). We evaluate gSDE both in simulation, on PyBullet continuous control tasks, and directly on three different real robots: a tendon-driven elastic robot, a quadruped and an RC car. The noise sampling interval of gSDE permits to have a compromise between performance and smoothness, which allows training directly on the real robots without loss of performance. The code is available at https://github.com/DLR-RM/stable-baselines3.