Context-Based Soft Actor Critic for Environments with Non-stationary Dynamics
This addresses the issue of non-stationary dynamics in reinforcement learning for researchers and practitioners, but it is incremental as it builds on existing Meta-RL and SAC methods.
The paper tackles the problem of deep reinforcement learning performance degrading in environments with non-stationary dynamics by proposing the Latent Context-based Soft Actor Critic (LC-SAC) method, which shows significantly better performance than SAC on MetaWorld ML1 tasks with drastic dynamics changes and comparable performance on MuJoCo tasks with stable dynamics.
The performance of deep reinforcement learning methods prone to degenerate when applied to environments with non-stationary dynamics. In this paper, we utilize the latent context recurrent encoders motivated by recent Meta-RL materials, and propose the Latent Context-based Soft Actor Critic (LC-SAC) method to address aforementioned issues. By minimizing the contrastive prediction loss function, the learned context variables capture the information of the environment dynamics and the recent behavior of the agent. Then combined with the soft policy iteration paradigm, the LC-SAC method alternates between soft policy evaluation and soft policy improvement until it converges to the optimal policy. Experimental results show that the performance of LC-SAC is significantly better than the SAC algorithm on the MetaWorld ML1 tasks whose dynamics changes drasticly among different episodes, and is comparable to SAC on the continuous control benchmark task MuJoCo whose dynamics changes slowly or doesn't change between different episodes. In addition, we also conduct relevant experiments to determine the impact of different hyperparameter settings on the performance of the LC-SAC algorithm and give the reasonable suggestions of hyperparameter setting.