Can Learned Optimization Make Reinforcement Learning Less Difficult?
This work addresses key challenges in reinforcement learning for decision-making applications, representing an incremental improvement by combining existing solutions into a learned optimizer.
The paper tackles the difficulties of non-stationarity, plasticity loss, and exploration in reinforcement learning by proposing OPEN, a meta-learned update rule, which outperforms or equals traditional optimizers in experiments on single and small sets of environments and shows strong generalization across diverse environments and agent architectures.
While reinforcement learning (RL) holds great potential for decision making in the real world, it suffers from a number of unique difficulties which often need specific consideration. In particular: it is highly non-stationary; suffers from high degrees of plasticity loss; and requires exploration to prevent premature convergence to local optima and maximize return. In this paper, we consider whether learned optimization can help overcome these problems. Our method, Learned Optimization for Plasticity, Exploration and Non-stationarity (OPEN), meta-learns an update rule whose input features and output structure are informed by previously proposed solutions to these difficulties. We show that our parameterization is flexible enough to enable meta-learning in diverse learning contexts, including the ability to use stochasticity for exploration. Our experiments demonstrate that when meta-trained on single and small sets of environments, OPEN outperforms or equals traditionally used optimizers. Furthermore, OPEN shows strong generalization characteristics across a range of environments and agent architectures.