Fine-Tuning without Performance Degradation
This addresses a key challenge in applying offline-learned policies in real-world domains, though it is incremental as it builds on prior fine-tuning improvements.
The paper tackles the problem of performance degradation during fine-tuning of offline-learned policies, showing that many existing algorithms suffer from either degradation or slow learning. It introduces a new algorithm based on Jump Start that reduces performance degradations and achieves faster fine-tuning compared to existing methods.
Fine-tuning policies learned offline remains a major challenge in application domains. Monotonic performance improvement during \emph{fine-tuning} is often challenging, as agents typically experience performance degradation at the early fine-tuning stage. The community has identified multiple difficulties in fine-tuning a learned network online, however, the majority of progress has focused on improving learning efficiency during fine-tuning. In practice, this comes at a serious cost during fine-tuning: initially, agent performance degrades as the agent explores and effectively overrides the policy learned offline. We show across a range of settings, many offline-to-online algorithms exhibit either (1) performance degradation or (2) slow learning (sometimes effectively no improvement) during fine-tuning. We introduce a new fine-tuning algorithm, based on an algorithm called Jump Start, that gradually allows more exploration based on online estimates of performance. Empirically, this approach achieves fast fine-tuning and significantly reduces performance degradations compared with existing algorithms designed to do the same.