A Max-Min Entropy Framework for Reinforcement Learning
This addresses exploration challenges in reinforcement learning for AI systems, representing a novel method rather than an incremental improvement.
The paper tackles the limitation of maximum entropy reinforcement learning by proposing a max-min entropy framework that promotes better exploration by learning to visit and maximize entropy in low-entropy states, resulting in drastic performance improvement over state-of-the-art RL algorithms.
In this paper, we propose a max-min entropy framework for reinforcement learning (RL) to overcome the limitation of the soft actor-critic (SAC) algorithm implementing the maximum entropy RL in model-free sample-based learning. Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future, the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration. For general Markov decision processes (MDPs), an efficient algorithm is constructed under the proposed max-min entropy framework based on disentanglement of exploration and exploitation. Numerical results show that the proposed algorithm yields drastic performance improvement over the current state-of-the-art RL algorithms.