$π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
This work addresses the problem of scaling RL for flow-based VLAs in robotics, enabling more efficient and effective fine-tuning for complex tasks, though it is incremental in improving existing methods.
The paper tackles the challenge of applying reinforcement learning (RL) to flow-based vision-language-action models, which have intractable action log-likelihoods, by introducing $\pi_{\text{RL}}$, a framework that implements two RL algorithms to enable efficient training in parallel simulation. It results in significant performance improvements, such as boosting models from 57.6% to 97.6% on LIBERO and from 41.6% to 85.7% on ManiSkill benchmarks.
Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying large-scale RL to flow-based VLAs (e.g., $π_0$, $π_{0.5}$) remains challenging due to intractable action log-likelihoods from iterative denoising. We address this challenge with $π_{\text{RL}}$, an open-source framework for training flow-based VLAs in parallel simulation. $π_{\text{RL}}$ implements two RL algorithms: (1) {Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) {Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate $π_{\text{RL}}$ on LIBERO and ManiSkill benchmarks. On LIBERO, $π_{\text{RL}}$ boosts few-shot SFT models $π_0$ and $π_{0.5}$ from 57.6% to 97.6% and from 77.1% to 98.3%, respectively. In ManiSkill, we train $π_{\text{RL}}$ in 320 parallel environments, improving $π_0$ from 41.6% to 85.7% and $π_{0.5}$ from 40.0% to 84.8% across 4352 pick-and-place tasks, demonstrating scalable multitask RL under heterogeneous simulation. Overall, $π_{\text{RL}}$ achieves significant performance gains and stronger generalization over SFT-models, validating the effectiveness of online RL for flow-based VLAs.