Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing
This work addresses the challenge of efficiently training diverse controllers for robots to adapt to damage, though it is incremental as it builds on existing CMA-MAE and approximation methods.
The paper tackled the problem of scaling Covariance Matrix Adaptation MAP-Annealing (CMA-MAE) to high-dimensional neural network controllers for robotic locomotion by proposing three efficient variants, which outperformed evolution strategies baselines and matched or exceeded state-of-the-art deep reinforcement learning methods in benchmark tasks.
Pre-training a diverse set of neural network controllers in simulation has enabled robots to adapt online to damage in robot locomotion tasks. However, finding diverse, high-performing controllers requires expensive network training and extensive tuning of a large number of hyperparameters. On the other hand, Covariance Matrix Adaptation MAP-Annealing (CMA-MAE), an evolution strategies (ES)-based quality diversity algorithm, does not have these limitations and has achieved state-of-the-art performance on standard QD benchmarks. However, CMA-MAE cannot scale to modern neural network controllers due to its quadratic complexity. We leverage efficient approximation methods in ES to propose three new CMA-MAE variants that scale to high dimensions. Our experiments show that the variants outperform ES-based baselines in benchmark robotic locomotion tasks, while being comparable with or exceeding state-of-the-art deep reinforcement learning-based quality diversity algorithms.