On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness
This work addresses the theoretical understanding of pre-training for RL generalization, providing foundational insights but is incremental as it builds on existing generalization frameworks.
The paper tackles the problem of generalization in reinforcement learning by theoretically analyzing the benefits of pre-training over training environments, showing that without target interaction, a near-optimal policy in an average sense is achievable, and with interaction, pre-training offers at most a constant asymptotic improvement but enables a non-asymptotic regret bound independent of state-action space.
Generalization in Reinforcement Learning (RL) aims to learn an agent during training that generalizes to the target environment. This paper studies RL generalization from a theoretical aspect: how much can we expect pre-training over training environments to be helpful? When the interaction with the target environment is not allowed, we certify that the best we can obtain is a near-optimal policy in an average sense, and we design an algorithm that achieves this goal. Furthermore, when the agent is allowed to interact with the target environment, we give a surprising result showing that asymptotically, the improvement from pre-training is at most a constant factor. On the other hand, in the non-asymptotic regime, we design an efficient algorithm and prove a distribution-based regret bound in the target environment that is independent of the state-action space.