The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective
This addresses a core challenge in reinforcement learning for improving agent efficiency and performance, though it appears incremental as it builds on existing entropy-based methods.
The paper tackles the exploration-exploitation imbalance in reinforcement learning by developing AdaZero, an adaptive framework based on entropy, which boosts final returns by up to fifteen times in challenging environments like Montezuma.
The imbalance of exploration and exploitation has long been a significant challenge in reinforcement learning. In policy optimization, excessive reliance on exploration reduces learning efficiency, while over-dependence on exploitation might trap agents in local optima. This paper revisits the exploration-exploitation dilemma from the perspective of entropy by revealing the relationship between entropy and the dynamic adaptive process of exploration and exploitation. Based on this theoretical insight, we establish an end-to-end adaptive framework called AdaZero, which automatically determines whether to explore or to exploit as well as their balance of strength. Experiments show that AdaZero significantly outperforms baseline models across various Atari and MuJoCo environments with only a single setting. Especially in the challenging environment of Montezuma, AdaZero boosts the final returns by up to fifteen times. Moreover, we conduct a series of visualization analyses to reveal the dynamics of our self-adaptive mechanism, demonstrating how entropy reflects and changes with respect to the agent's performance and adaptive process.