Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models
This work addresses the challenge of enhancing action accuracy in VLA models for robotic manipulation, representing an incremental improvement by integrating RL into the post-training paradigm.
The paper tackles the problem of unsatisfactory action accuracy in Vision-Language-Action (VLA) flow models for robotic manipulation by proposing an offline reinforcement learning (RL) post-training method called Adaptive Reinforced Flow Matching (ARFM), which adaptively balances RL advantage and flow loss variance to achieve more stable and efficient fine-tuning, resulting in improved generalization, robustness, few-shot learning, and continuous learning performance.
Vision-Language-Action (VLA) models based on flow matching have shown excellent performance in general-purpose robotic manipulation tasks. However, the action accuracy of these models on complex downstream tasks is unsatisfactory. One important reason is that these models rely solely on the post-training paradigm of imitation learning, which makes it difficult to have a deeper understanding of the distribution properties of data quality, which is exactly what Reinforcement Learning (RL) excels at. In this paper, we theoretically propose an offline RL post-training objective for VLA flow models and induce an efficient and feasible offline RL fine-tuning algorithm -- Adaptive Reinforced Flow Matching (ARFM). By introducing an adaptively adjusted scaling factor in the VLA flow model loss, we construct a principled bias-variance trade-off objective function to optimally control the impact of RL signal on flow loss. ARFM adaptively balances RL advantage preservation and flow loss gradient variance control, resulting in a more stable and efficient fine-tuning process. Extensive simulation and real-world experimental results show that ARFM exhibits excellent generalization, robustness, few-shot learning, and continuous learning performance.