End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions
This work solves a fundamental problem in reinforcement learning for researchers and practitioners by enabling efficient learning in linear Bellman complete MDPs with deterministic transitions, representing an incremental advance over prior computationally efficient algorithms.
The paper tackles the problem of reinforcement learning with linear function approximation in Markov Decision Processes that satisfy linear Bellman completeness, providing a computationally efficient algorithm for deterministic transitions, stochastic initial states, and rewards. It achieves an ε-optimal policy with sample and computational complexity polynomial in the horizon, feature dimension, and 1/ε, addressing limitations of prior methods that were restricted to small action spaces or required strong oracle assumptions.
We study reinforcement learning (RL) with linear function approximation in Markov Decision Processes (MDPs) satisfying \emph{linear Bellman completeness} -- a fundamental setting where the Bellman backup of any linear value function remains linear. While statistically tractable, prior computationally efficient algorithms are either limited to small action spaces or require strong oracle assumptions over the feature space. We provide a computationally efficient algorithm for linear Bellman complete MDPs with \emph{deterministic transitions}, stochastic initial states, and stochastic rewards. For finite action spaces, our algorithm is end-to-end efficient; for large or infinite action spaces, we require only a standard argmax oracle over actions. Our algorithm learns an $\varepsilon$-optimal policy with sample and computational complexity polynomial in the horizon, feature dimension, and $1/\varepsilon$.