Controllable Flow Matching for Online Reinforcement Learning
This addresses stability and efficiency challenges in online reinforcement learning for robotics and control tasks, representing an incremental improvement over existing MBRL approaches.
The paper tackles the problem of model error accumulation in model-based reinforcement learning by proposing CtrlFlow, a trajectory-level synthetic method using conditional flow matching that directly models high-return trajectories without explicit environment dynamics. The method achieves better performance on MuJoCo benchmarks than dynamics models and shows superior sample efficiency compared to standard MBRL methods.
Model-based reinforcement learning (MBRL) typically relies on modeling environment dynamics for data efficiency. However, due to the accumulation of model errors over long-horizon rollouts, such methods often face challenges in maintaining modeling stability. To address this, we propose CtrlFlow, a trajectory-level synthetic method using conditional flow matching (CFM), which directly modeling the distribution of trajectories from initial states to high-return terminal states without explicitly modeling the environment transition function. Our method ensures optimal trajectory sampling by minimizing the control energy governed by the non-linear Controllability Gramian Matrix, while the generated diverse trajectory data significantly enhances the robustness and cross-task generalization of policy learning. In online settings, CtrlFlow demonstrates the better performance on common MuJoCo benchmark tasks than dynamics models and achieves superior sample efficiency compared to standard MBRL methods.