Simple Noisy Environment Augmentation for Reinforcement Learning
This work addresses the challenge of data diversity and exploration in RL for practitioners, though it is incremental as it builds on existing augmentation techniques by extending them to broader RL contexts.
The paper tackles the problem of improving reinforcement learning (RL) performance by introducing generic wrappers that augment environments with noise in states, rewards, and transitions, resulting in enhanced agent exploration and training data diversity, with experimental validation across three RL algorithms and five MuJoCo environments.
Data augmentation is a widely used technique for improving model performance in machine learning, particularly in computer vision and natural language processing. Recently, there has been increasing interest in applying augmentation techniques to reinforcement learning (RL) problems, with a focus on image-based augmentation. In this paper, we explore a set of generic wrappers designed to augment RL environments with noise and encourage agent exploration and improve training data diversity which are applicable to a broad spectrum of RL algorithms and environments. Specifically, we concentrate on augmentations concerning states, rewards, and transition dynamics and introduce two novel augmentation techniques. In addition, we introduce a noise rate hyperparameter for control over the frequency of noise injection. We present experimental results on the impact of these wrappers on return using three popular RL algorithms, Soft Actor-Critic (SAC), Twin Delayed DDPG (TD3), and Proximal Policy Optimization (PPO), across five MuJoCo environments. To support the choice of augmentation technique in practice, we also present analysis that explores the performance these techniques across environments. Lastly, we publish the wrappers in our noisyenv repository for use with gym environments.