CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning
This addresses the exploration problem in reinforcement learning for researchers and practitioners, offering a simple, efficient solution with theoretical guarantees, though it appears incremental as it builds on existing bandit techniques and value networks.
The paper tackles the challenge of exploration in deep reinforcement learning by introducing CAE, a lightweight algorithm that repurposes value networks for exploration without extra parameters, achieving provable regret bounds and outperforming state-of-the-art baselines in experiments on MuJoCo and MiniHack.
Exploration remains a critical challenge in reinforcement learning, as many existing methods either lack theoretical guarantees or fall short of practical effectiveness. In this paper, we introduce CAE, a lightweight algorithm that repurposes the value networks in standard deep RL algorithms to drive exploration without introducing additional parameters. CAE utilizes any linear multi-armed bandit technique and incorporates an appropriate scaling strategy, enabling efficient exploration with provable sub-linear regret bounds and practical stability. Notably, it is simple to implement, requiring only around 10 lines of code. In complex tasks where learning an effective value network proves challenging, we propose CAE+, an extension of CAE that incorporates an auxiliary network. This extension increases the parameter count by less than 1% while maintaining implementation simplicity, adding only about 10 additional lines of code. Experiments on MuJoCo and MiniHack show that both CAE and CAE+ outperform state-of-the-art baselines, bridging the gap between theoretical rigor and practical efficiency.