Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning
This addresses stability and scalability problems in deep RL for continuous control, though it appears incremental as it builds on existing maximum entropy RL frameworks.
The paper tackles instability and scalability issues in maximum entropy deep reinforcement learning for continuous control tasks by proposing the Deep Soft Policy Gradient (DSPG) algorithm, which outperforms prior off-policy methods in experiments.
Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid locomotion. Besides, the optimality of desired Boltzmann policy set for non-optimal soft value function is not persuasive enough. In this paper, we first derive soft policy gradient based on entropy regularized expected reward objective for RL with continuous actions. Then, we present an off-policy actor-critic, model-free maximum entropy deep RL algorithm called deep soft policy gradient (DSPG) by combining soft policy gradient with soft Bellman equation. To ensure stable learning while eliminating the need of two separate critics for soft value functions, we leverage double sampling approach to making the soft Bellman equation tractable. The experimental results demonstrate that our method outperforms in performance over off-policy prior methods.