ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation
This addresses the scalability issue in imitation learning for bimanual manipulation, though it is incremental as it builds on existing data augmentation and diffusion models.
The paper tackles the problem of costly and time-consuming collection of diverse real-world demonstrations for training robust bimanual manipulation policies by proposing ROPA, a method that fine-tunes Stable Diffusion to synthesize third-person RGB and RGB-D observations of novel robot poses with corresponding joint-space action labels, and it outperforms baselines in 2625 simulation and 300 real-world trials.
Training robust bimanual manipulation policies via imitation learning requires demonstration data with broad coverage over robot poses, contacts, and scene contexts. However, collecting diverse and precise real-world demonstrations is costly and time-consuming, which hinders scalability. Prior works have addressed this with data augmentation, typically for either eye-in-hand (wrist camera) setups with RGB inputs or for generating novel images without paired actions, leaving augmentation for eye-to-hand (third-person) RGB-D training with new action labels less explored. In this paper, we propose Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation (ROPA), an offline imitation learning data augmentation method that fine-tunes Stable Diffusion to synthesize third-person RGB and RGB-D observations of novel robot poses. Our approach simultaneously generates corresponding joint-space action labels while employing constrained optimization to enforce physical consistency through appropriate gripper-to-object contact constraints in bimanual scenarios. We evaluate our method on 5 simulated and 3 real-world tasks. Our results across 2625 simulation trials and 300 real-world trials demonstrate that ROPA outperforms baselines and ablations, showing its potential for scalable RGB and RGB-D data augmentation in eye-to-hand bimanual manipulation. Our project website is available at: https://ropaaug.github.io/.