Flow Matching Policy with Entropy Regularization
This work addresses computational bottlenecks in reinforcement learning for robotics, offering a more efficient and principled approach to policy optimization, though it appears incremental as it builds on existing flow matching and entropy regularization concepts.
The paper tackled the computational inefficiency and indirect entropy control in diffusion-based reinforcement learning policies by proposing FMER, an ODE-based framework that uses flow matching and entropy regularization, achieving faster training times (7x reduction compared to heavy baselines) and outperforming state-of-the-art methods on sparse multi-goal benchmarks.
Diffusion-based policies have gained significant popularity in Reinforcement Learning (RL) due to their ability to represent complex, non-Gaussian distributions. Stochastic Differential Equation (SDE)-based diffusion policies often rely on indirect entropy control due to the intractability of the exact entropy, while also suffering from computationally prohibitive policy gradients through the iterative denoising chain. To overcome these issues, we propose Flow Matching Policy with Entropy Regularization (FMER), an Ordinary Differential Equation (ODE)-based online RL framework. FMER parameterizes the policy via flow matching and samples actions along a straight probability path, motivated by optimal transport. FMER leverages the model's generative nature to construct an advantage-weighted target velocity field from a candidate set, steering policy updates toward high-value regions. By deriving a tractable entropy objective, FMER enables principled maximum-entropy optimization for enhanced exploration. Experiments on sparse multi-goal FrankaKitchen benchmarks demonstrate that FMER outperforms state-of-the-art methods, while remaining competitive on standard MuJoco benchmarks. Moreover, FMER reduces training time by 7x compared to heavy diffusion baselines (QVPO) and 10-15% relative to efficient variants.