Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings
This work addresses the challenge of scaling reinforcement learning to large or continuous POMDPs, which is crucial for applications in robotics and AI, though it relies on specific assumptions like deterministic latent transitions and action gaps.
The paper tackles the problem of reinforcement learning in large-scale partially observable Markov decision processes (POMDPs) by proposing an algorithm that finds the exact optimal policy with computational and statistical complexities scaling polynomially with horizon and feature dimension, avoiding exponential dependencies on state and observation space sizes.
We study reinforcement learning with function approximation for large-scale Partially Observable Markov Decision Processes (POMDPs) where the state space and observation space are large or even continuous. Particularly, we consider Hilbert space embeddings of POMDP where the feature of latent states and the feature of observations admit a conditional Hilbert space embedding of the observation emission process, and the latent state transition is deterministic. Under the function approximation setup where the optimal latent state-action $Q$-function is linear in the state feature, and the optimal $Q$-function has a gap in actions, we provide a \emph{computationally and statistically efficient} algorithm for finding the \emph{exact optimal} policy. We show our algorithm's computational and statistical complexities scale polynomially with respect to the horizon and the intrinsic dimension of the feature on the observation space. Furthermore, we show both the deterministic latent transitions and gap assumptions are necessary to avoid statistical complexity exponential in horizon or dimension. Since our guarantee does not have an explicit dependence on the size of the state and observation spaces, our algorithm provably scales to large-scale POMDPs.