Improved Worst-Case Regret Bounds for Randomized Least-Squares Value Iteration
This work addresses regret minimization for researchers in reinforcement learning, providing an incremental improvement in worst-case bounds for a specific algorithm.
The paper tackles the problem of minimizing worst-case regret in reinforcement learning by introducing a clipping variant of randomized least-squares value iteration, achieving a regret bound of $ ilde{\mathrm{O}}(H^2S\sqrt{AT})$ that matches state-of-the-art bounds for Thompson Sampling-based methods.
This paper studies regret minimization with randomized value functions in reinforcement learning. In tabular finite-horizon Markov Decision Processes, we introduce a clipping variant of one classical Thompson Sampling (TS)-like algorithm, randomized least-squares value iteration (RLSVI). Our $\tilde{\mathrm{O}}(H^2S\sqrt{AT})$ high-probability worst-case regret bound improves the previous sharpest worst-case regret bounds for RLSVI and matches the existing state-of-the-art worst-case TS-based regret bounds.