Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation
For reinforcement learning practitioners, this work provides theoretical insights and an algorithm for efficiently adapting offline pretrained value functions with limited online interaction, though the improvement is conditional on a structural assumption.
The paper studies offline-to-online reinforcement learning under general function approximation, establishing a minimax lower bound showing that even with a near-optimal pretrained Q-function, online adaptation can be no more efficient than pure online RL on hard instances. Under a novel structural condition, they propose O2O-LSVI with provably improved sample complexity over pure online RL, and validate with neural-network experiments.
We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained $Q$-function, the learner aims to adapt it to the target environment using only a limited amount of online interaction. We first characterize the difficulty of this setting by establishing a minimax lower bound, showing that even when the pretrained $Q$-function is close to optimal $Q^\star$, online adaptation can be no more efficient than pure online RL on certain hard instances. On the positive side, under a novel structural condition on the offline-pretrained value functions, we propose O2O-LSVI, an adaptation algorithm with problem-dependent sample complexity that provably improves over pure online RL. Finally, we complement our theory with neural-network experiments that demonstrate the practical effectiveness of the proposed method.