The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning
This addresses the problem of hyperparameter sensitivity in RL algorithm evaluation for researchers, though it is incremental as it builds on existing benchmarking methods.
The paper introduces the Cross-environment Hyperparameter Setting Benchmark to compare RL algorithms across environments with a single hyperparameter setting, demonstrating robustness to noise and low computational cost, and uses it to show no meaningful performance difference between Ornstein-Uhlenbeck and Gaussian noise in DDPG on the DM Control suite.
This paper introduces a new empirical methodology, the Cross-environment Hyperparameter Setting Benchmark, that compares RL algorithms across environments using a single hyperparameter setting, encouraging algorithmic development which is insensitive to hyperparameters. We demonstrate that this benchmark is robust to statistical noise and obtains qualitatively similar results across repeated applications, even when using few samples. This robustness makes the benchmark computationally cheap to apply, allowing statistically sound insights at low cost. We demonstrate two example instantiations of the CHS, on a set of six small control environments (SC-CHS) and on the entire DM Control suite of 28 environments (DMC-CHS). Finally, to illustrate the applicability of the CHS to modern RL algorithms on challenging environments, we conduct a novel empirical study of an open question in the continuous control literature. We show, with high confidence, that there is no meaningful difference in performance between Ornstein-Uhlenbeck noise and uncorrelated Gaussian noise for exploration with the DDPG algorithm on the DMC-CHS.