Entropy Augmented Reinforcement Learning
This work addresses exploration limitations in reinforcement learning for researchers and practitioners, though it appears incremental as it builds on existing entropy-based methods like SAC.
The paper tackles the issue of constrained exploration in trust region reinforcement learning methods by proposing an entropy augmentation technique that integrates with on-policy algorithms, resulting in improved performance on MuJoCo benchmarks and enhanced exploration in custom environments.
Deep reinforcement learning was instigated with the presence of trust region methods, being scalable and efficient. However, the pessimism of such algorithms, among which it forces to constrain in a trust region by all means, has been proven to suppress the exploration and harm the performance. Exploratory algorithm such as SAC, while utilizes the entropy to encourage exploration, implicitly optimizing another objective yet. We first observed this inconsistency, and therefore put forward an analogous augmentation technique, which combines well with the on-policy algorithms, when a value critic is involved. Surprisingly, the proposed method consistently satisfies the soft policy improvement theorem, while being more extensible. As the analysis advises, it is crucial to control the temperature coefficient to balance the exploration and exploitation. Empirical tests on MuJoCo benchmark tasks show that the agent is heartened towards higher reward regions, and enjoys a finer performance. Furthermore, we verify the exploration bonus of our method on a set of custom environments.