Minimax Optimal Reinforcement Learning with Quasi-Optimism
This addresses the challenge of balancing theoretical guarantees with practical usability in reinforcement learning, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the problem of developing a reinforcement learning algorithm that is both practical and provably optimal by introducing EQO, which achieves the sharpest known regret bound for tabular RL under mild assumptions and outperforms existing algorithms in regret and computational efficiency.
In our quest for a reinforcement learning (RL) algorithm that is both practical and provably optimal, we introduce EQO (Exploration via Quasi-Optimism). Unlike existing minimax optimal approaches, EQO avoids reliance on empirical variances and employs a simple bonus term proportional to the inverse of the state-action visit count. Central to EQO is the concept of quasi-optimism, where estimated values need not be fully optimistic, allowing for a simpler yet effective exploration strategy. The algorithm achieves the sharpest known regret bound for tabular RL under the mildest assumptions, proving that fast convergence can be attained with a practical and computationally efficient approach. Empirical evaluations demonstrate that EQO consistently outperforms existing algorithms in both regret performance and computational efficiency, providing the best of both theoretical soundness and practical effectiveness.