Double Meta-Learning for Data Efficient Policy Optimization in Non-Stationary Environments
This work addresses the problem of data-efficient policy learning in environments where dynamics change over time, which is crucial for applications requiring rapid adaptation with limited real-world interaction.
This paper tackles data-efficient policy optimization in non-stationary environments by proposing a meta-reinforcement learning approach. It simultaneously trains a meta-model of the environment and a meta-policy, using the converged meta-learned dynamic model to generate simulated data for policy optimization. The method achieves data efficiency comparable to model-based learning while maintaining the high asymptotic performance of model-free meta-reinforcement learning.
We are interested in learning models of non-stationary environments, which can be framed as a multi-task learning problem. Model-free reinforcement learning algorithms can achieve good asymptotic performance in multi-task learning at a cost of extensive sampling, due to their approach, which requires learning from scratch. While model-based approaches are among the most data efficient learning algorithms, they still struggle with complex tasks and model uncertainties. Meta-reinforcement learning addresses the efficiency and generalization challenges on multi task learning by quickly leveraging the meta-prior policy for a new task. In this paper, we propose a meta-reinforcement learning approach to learn the dynamic model of a non-stationary environment to be used for meta-policy optimization later. Due to the sample efficiency of model-based learning methods, we are able to simultaneously train both the meta-model of the non-stationary environment and the meta-policy until dynamic model convergence. Then, the meta-learned dynamic model of the environment will generate simulated data for meta-policy optimization. Our experiment demonstrates that our proposed method can meta-learn the policy in a non-stationary environment with the data efficiency of model-based learning approaches while achieving the high asymptotic performance of model-free meta-reinforcement learning.