Provably Efficient and Agile Randomized Q-Learning
This work addresses the need for more efficient and responsive algorithms in reinforcement learning, offering a provably effective solution with potential applications in domains requiring agile decision-making.
The paper tackles the problem of limited theoretical understanding and computational inefficiency in model-free reinforcement learning by proposing RandomizedQ, a Q-learning variant with sampling-based exploration and step-wise policy updates, achieving an O(√(H^5SAT)) regret bound and demonstrating superior empirical performance on benchmarks.
While Bayesian-based exploration often demonstrates superior empirical performance compared to bonus-based methods in model-based reinforcement learning (RL), its theoretical understanding remains limited for model-free settings. Existing provable algorithms either suffer from computational intractability or rely on stage-wise policy updates which reduce responsiveness and slow down the learning process. In this paper, we propose a novel variant of Q-learning algorithm, refereed to as RandomizedQ, which integrates sampling-based exploration with agile, step-wise, policy updates, for episodic tabular RL. We establish an $\widetilde{O}(\sqrt{H^5SAT})$ regret bound, where $S$ is the number of states, $A$ is the number of actions, $H$ is the episode length, and $T$ is the total number of episodes. In addition, we present a logarithmic regret bound under a mild positive sub-optimality condition on the optimal Q-function. Empirically, RandomizedQ exhibits outstanding performance compared to existing Q-learning variants with both bonus-based and Bayesian-based exploration on standard benchmarks.