Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models
This work addresses the challenge of automating algorithm design in reinforcement learning, offering a novel but incremental approach that could benefit researchers and practitioners in AI.
The authors tackled the problem of manually designing reinforcement learning algorithms by developing an evolutionary framework that discovers executable update rules, achieving competitive performance on Gymnasium benchmarks compared to established baselines like SAC, PPO, DQN, and A2C.
Reinforcement learning algorithms are defined by their learning update rules, which are typically hand-designed and fixed. We present an evolutionary framework for discovering reinforcement learning algorithms by searching directly over executable update rules that implement complete training procedures. The approach builds on REvolve, an evolutionary system that uses large language models as generative variation operators, and extends it from reward-function discovery to algorithm discovery. To promote the emergence of nonstandard learning rules, the search excludes canonical mechanisms such as actor--critic structures, temporal-difference losses, and value bootstrapping. Because reinforcement learning algorithms are highly sensitive to internal scalar parameters, we introduce a post-evolution refinement stage in which a large language model proposes feasible hyperparameter ranges for each evolved update rule. Evaluated end-to-end by full training runs on multiple Gymnasium benchmarks, the discovered algorithms achieve competitive performance relative to established baselines, including SAC, PPO, DQN, and A2C.