Improved No-Regret Algorithms for Stochastic Shortest Path with Linear MDP
This provides improved algorithms for reinforcement learning in stochastic shortest path settings, addressing computational efficiency and regret bounds for researchers and practitioners in the field.
The paper tackles the stochastic shortest path problem with linear MDPs by introducing two new no-regret algorithms that improve over prior work, achieving a computationally efficient regret bound of $\widetilde{O}\left(\sqrt{d^3B_{\star}^2T_{\star} K} ight)$ and a horizon-free bound of $\widetilde{O}(d^{3.5}B_{\star}\sqrt{K})$ that nearly matches a lower bound.
We introduce two new no-regret algorithms for the stochastic shortest path (SSP) problem with a linear MDP that significantly improve over the only existing results of (Vial et al., 2021). Our first algorithm is computationally efficient and achieves a regret bound $\widetilde{O}\left(\sqrt{d^3B_{\star}^2T_{\star} K}\right)$, where $d$ is the dimension of the feature space, $B_{\star}$ and $T_{\star}$ are upper bounds of the expected costs and hitting time of the optimal policy respectively, and $K$ is the number of episodes. The same algorithm with a slight modification also achieves logarithmic regret of order $O\left(\frac{d^3B_{\star}^4}{c_{\min}^2\text{gap}_{\min}}\ln^5\frac{dB_{\star} K}{c_{\min}} \right)$, where $\text{gap}_{\min}$ is the minimum sub-optimality gap and $c_{\min}$ is the minimum cost over all state-action pairs. Our result is obtained by developing a simpler and improved analysis for the finite-horizon approximation of (Cohen et al., 2021) with a smaller approximation error, which might be of independent interest. On the other hand, using variance-aware confidence sets in a global optimization problem, our second algorithm is computationally inefficient but achieves the first "horizon-free" regret bound $\widetilde{O}(d^{3.5}B_{\star}\sqrt{K})$ with no polynomial dependency on $T_{\star}$ or $1/c_{\min}$, almost matching the $Ω(dB_{\star}\sqrt{K})$ lower bound from (Min et al., 2021).