Constant in an Ever-Changing World
This addresses instability issues in reinforcement learning for practitioners, but it appears incremental as it builds on conventional algorithms.
The paper tackles the problem of oscillations and instability in reinforcement learning training by proposing the CIC framework, which enhances algorithmic stability and improves performance on five MuJoCo environments without additional computational cost.
The training process of reinforcement learning often suffers from severe oscillations, leading to instability and degraded performance. In this paper, we propose a Constant in an Ever-Changing World (CIC) framework that enhances algorithmic stability to improve performance. CIC maintains both a representative policy and a current policy. Instead of updating the representative policy blindly, CIC selectively updates it only when the current policy demonstrates superiority. Furthermore, CIC employs an adaptive adjustment mechanism, enabling the representative and current policies to jointly facilitate critic training. We evaluate CIC on five MuJoCo environments, and the results show that CIC improves the performance of conventional algorithms without incurring additional computational cost.