Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration
This work addresses the challenge of efficient exploration in reinforcement learning for researchers, offering theoretical improvements over existing methods but is incremental in nature.
The paper tackles the problem of provably efficient learning in MDPs with linear function approximation under a standard low inherent Bellman error assumption, showing that a computationally tractable reward-free algorithm can learn near-optimal policies for any linear reward function with strong PAC guarantees.
There has been growing progress on theoretical analyses for provably efficient learning in MDPs with linear function approximation, but much of the existing work has made strong assumptions to enable exploration by conventional exploration frameworks. Typically these assumptions are stronger than what is needed to find good solutions in the batch setting. In this work, we show how under a more standard notion of low inherent Bellman error, typically employed in least-square value iteration-style algorithms, we can provide strong PAC guarantees on learning a near optimal value function provided that the linear space is sufficiently "explorable". We present a computationally tractable algorithm for the reward-free setting and show how it can be used to learn a near optimal policy for any (linear) reward function, which is revealed only once learning has completed. If this reward function is also estimated from the samples gathered during pure exploration, our results also provide same-order PAC guarantees on the performance of the resulting policy for this setting.