Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
This work addresses the problem of costly human demonstrations for fine-tuning large vision-language-action models, offering a scalable self-improvement method for robotics and AI applications, though it is incremental as it builds on existing fine-tuning and RL techniques.
The paper tackles the scalability and generalization limitations of supervised fine-tuning for vision-language-action models by introducing a three-stage framework (PLD) that uses residual reinforcement learning and distribution-aware data collection to improve model performance, achieving near-saturated 99% task success on LIBERO, over 50% gains in SimplerEnv, and 100% success on real-world manipulation tasks.
Supervised fine-tuning (SFT) has become the de facto post-training strategy for large vision-language-action (VLA) models, but its reliance on costly human demonstrations limits scalability and generalization. We propose Probe, Learn, Distill (PLD), a three-stage plug-and-play framework that improves VLAs through residual reinforcement learning (RL) and distribution-aware data collection. In Stage 1, we train lightweight residual actors to probe failure regions of the VLA generalist. In Stage 2, we use a hybrid rollout scheme that aligns collected trajectories with the generalist's deployment distribution while capturing recovery behaviors. In Stage 3, we distill the curated trajectories back into the generalist with standard SFT. PLD achieves near-saturated 99% task success on LIBERO, over 50% gains in SimplerEnv, and 100% success on real-world Franka and YAM arm manipulation tasks. Ablations show that residual probing and distribution-aware replay are key to collecting deployment-aligned data that improves both seen and unseen tasks, offering a scalable path toward self-improving VLA models.