Towards Automatic Actor-Critic Solutions to Continuous Control
This work addresses the problem of computational expense and difficulty in applying actor-critic methods to new domains for researchers and practitioners, though it is incremental as it builds upon the Soft Actor-Critic algorithm.
The paper tackled the difficulty of applying model-free off-policy actor-critic methods to new domains due to reliance on design tricks and hyperparameters, by creating an evolutionary approach that automatically tunes these decisions and eliminates RL-specific hyperparameters from Soft Actor-Critic, resulting in outperforming well-tuned hyperparameter settings in benchmarks like the DeepMind Control Suite.
Model-free off-policy actor-critic methods are an efficient solution to complex continuous control tasks. However, these algorithms rely on a number of design tricks and hyperparameters, making their application to new domains difficult and computationally expensive. This paper creates an evolutionary approach that automatically tunes these design decisions and eliminates the RL-specific hyperparameters from the Soft Actor-Critic algorithm. Our design is sample efficient and provides practical advantages over baseline approaches, including improved exploration, generalization over multiple control frequencies, and a robust ensemble of high-performance policies. Empirically, we show that our agent outperforms well-tuned hyperparameter settings in popular benchmarks from the DeepMind Control Suite. We then apply it to less common control tasks outside of simulated robotics to find high-performance solutions with minimal compute and research effort.