A View of the Certainty-Equivalence Method for PAC RL as an Application of the Trajectory Tree Method
This work provides theoretical insights and improved bounds for PAC RL algorithms, benefiting researchers in reinforcement learning theory, but it is incremental as it builds on existing methods like CEM and TTM.
The paper tackles the problem of analyzing the certainty-equivalence method (CEM) in PAC reinforcement learning by viewing it as an application of the trajectory tree method (TTM), resulting in new proofs, weaker reward assumptions, and improved sample-complexity upper bounds, with a 40% reduction in sample complexity for small mistake probabilities in non-stationary MDPs.
Reinforcement learning (RL) enables an agent interacting with an unknown MDP $M$ to optimise its behaviour by observing transitions sampled from $M$. A natural entity that emerges in the agent's reasoning is $\widehat{M}$, the maximum likelihood estimate of $M$ based on the observed transitions. The well-known \textit{certainty-equivalence} method (CEM) dictates that the agent update its behaviour to $\widehatπ$, which is an optimal policy for $\widehat{M}$. Not only is CEM intuitive, it has been shown to enjoy minimax-optimal sample complexity in some regions of the parameter space for PAC RL with a generative model~\citep{Agarwal2020GenModel}. A seemingly unrelated algorithm is the ``trajectory tree method'' (TTM)~\citep{Kearns+MN:1999}, originally developed for efficient decision-time planning in large POMDPs. This paper presents a theoretical investigation that stems from the surprising finding that CEM may indeed be viewed as an application of TTM. The qualitative benefits of this view are (1) new and simple proofs of sample complexity upper bounds for CEM, in fact under a (2) weaker assumption on the rewards than is prevalent in the current literature. Our analysis applies to both non-stationary and stationary MDPs. Quantitatively, we obtain (3) improvements in the sample-complexity upper bounds for CEM both for non-stationary and stationary MDPs, in the regime that the ``mistake probability'' $δ$ is small. Additionally, we show (4) a lower bound on the sample complexity for finite-horizon MDPs, which establishes the minimax-optimality of our upper bound for non-stationary MDPs in the small-$δ$ regime.