Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
This addresses the problem of sample efficiency in offline-to-online RL for AI systems, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the challenge of extending generative model policies from offline to online reinforcement learning by proposing FINO, which injects noise into flow matching-based policies to enhance exploration, achieving superior performance in diverse tasks under limited online budgets.
Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.