Soft policy optimization using dual-track advantage estimator
This work addresses the problem of slow training and local optima in on-policy RL algorithms for researchers and practitioners, representing an incremental improvement over existing methods.
The paper tackles the challenge of balancing exploration and exploitation in reinforcement learning by softening proximal policy optimization with entropy and a dynamic temperature coefficient, and introduces a dual-track advantage estimator to accelerate convergence. The method significantly speeds up training and achieves state-of-the-art cumulative returns on the Mujoco environment.
In reinforcement learning (RL), we always expect the agent to explore as many states as possible in the initial stage of training and exploit the explored information in the subsequent stage to discover the most returnable trajectory. Based on this principle, in this paper, we soften the proximal policy optimization by introducing the entropy and dynamically setting the temperature coefficient to balance the opportunity of exploration and exploitation. While maximizing the expected reward, the agent will also seek other trajectories to avoid the local optimal policy. Nevertheless, the increase of randomness induced by entropy will reduce the train speed in the early stage. Integrating the temporal-difference (TD) method and the general advantage estimator (GAE), we propose the dual-track advantage estimator (DTAE) to accelerate the convergence of value functions and further enhance the performance of the algorithm. Compared with other on-policy RL algorithms on the Mujoco environment, the proposed method not only significantly speeds up the training but also achieves the most advanced results in cumulative return.