Provably Efficient Model-based Policy Adaptation
This addresses the problem of efficient policy adaptation for reinforcement learning practitioners, offering a provably efficient method that works on out-of-distribution environments, though it builds incrementally on existing ideas.
The paper tackles the high sample complexity of reinforcement learning by proposing a model-based approach for adapting pre-trained policies to new environments, achieving state-of-the-art performance with significantly lower sample complexity in continuous control tasks.
The high sample complexity of reinforcement learning challenges its use in practice. A promising approach is to quickly adapt pre-trained policies to new environments. Existing methods for this policy adaptation problem typically rely on domain randomization and meta-learning, by sampling from some distribution of target environments during pre-training, and thus face difficulty on out-of-distribution target environments. We propose new model-based mechanisms that are able to make online adaptation in unseen target environments, by combining ideas from no-regret online learning and adaptive control. We prove that the approach learns policies in the target environment that can quickly recover trajectories from the source environment, and establish the rate of convergence in general settings. We demonstrate the benefits of our approach for policy adaptation in a diverse set of continuous control tasks, achieving the performance of state-of-the-art methods with much lower sample complexity.