Policy Optimization Achieves Data-Dependent Regret Bounds in MDPs with Unknown Transitions
It resolves the open problem of whether policy optimization can achieve data-dependent guarantees under unknown transitions, providing the first such results for tabular MDPs.
The paper develops a policy optimization algorithm for online episodic tabular MDPs with unknown transitions that achieves data-dependent regret bounds, including first-order, second-order, and path-length bounds, while also attaining gap-dependent polylog(T) regret in stochastic regimes.
We study policy optimization for online episodic tabular Markov decision processes with unknown transition kernels, aiming for best-of-both-worlds guarantees together with data-dependent regret bounds. Recent work (Dann et al., 2023; Li et al., 2026) has shown that policy optimization can adapt to both adversarial and stochastic losses with first-order, second-order, and path-length bounds, but only under known transitions, leaving open whether such data-dependent guarantees are achievable by policy optimization when the transition kernel is unknown. We resolve this by developing a new algorithm based on optimistic follow-the-regularized-leader that attains these guarantees under unknown transitions. The key ingredient is a new design of optimistic $Q$-function estimators together with a data-dependent transition bonus that controls estimator bias through the loss-prediction error. Our analysis further identifies an unavoidable transition-dependent complexity term that captures the intrinsic cost of estimating the transition kernel. As a result, we obtain first-order, second-order, and path-length bounds with the transition-dependent complexity term while simultaneously achieving gap-dependent $\mathrm{polylog}(T)$ regret in the stochastic regime.