The Three Regimes of Offline-to-Online Reinforcement Learning
This provides a principled framework for researchers and practitioners in RL to improve fine-tuning strategies, though it is incremental as it builds on existing paradigms.
The paper tackles the inconsistency in offline-to-online reinforcement learning by proposing a stability-plasticity principle to guide design choices, validating it with a large-scale study that aligns in 45 of 63 cases.
Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online-fine tuning that work well in one setting can fail completely in another. We propose a stability--plasticity principle that can explain this inconsistency: we should preserve the knowledge of pretrained policy or offline dataset during online fine-tuning, whichever is better, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 of 63 cases. This work provides a principled framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.