Revisiting Discrete Soft Actor-Critic
This work solves performance instability in discrete reinforcement learning for applications such as gaming, but it is incremental as it adapts an existing continuous method to discrete settings.
The paper tackled adapting Soft Actor-Critic (SAC) to discrete action spaces by addressing Q value underestimation and instability issues, proposing Stable Discrete SAC (SDSAC) which improved performance on benchmarks like Atari games and a large-scale MOBA game.
We study the adaption of Soft Actor-Critic (SAC), which is considered as a state-of-the-art reinforcement learning (RL) algorithm, from continuous action space to discrete action space. We revisit vanilla discrete SAC and provide an in-depth understanding of its Q value underestimation and performance instability issues when applied to discrete settings. We thereby propose Stable Discrete SAC (SDSAC), an algorithm that leverages entropy-penalty and double average Q-learning with Q-clip to address these issues. Extensive experiments on typical benchmarks with discrete action space, including Atari games and a large-scale MOBA game, show the efficacy of our proposed method. Our code is at: https://github.com/coldsummerday/SD-SAC.git.