Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning
This work addresses the challenge of visual generalization for robots in open-world environments, representing an incremental improvement with strong specific gains.
The paper tackles the problem of enabling visuomotor robots to generalize across diverse open-world scenarios by proposing Maniwhere, a visual reinforcement learning framework that significantly outperforms state-of-the-art methods on 8 manipulation tasks across 3 hardware platforms.
Can we endow visuomotor robots with generalization capabilities to operate in diverse open-world scenarios? In this paper, we propose \textbf{Maniwhere}, a generalizable framework tailored for visual reinforcement learning, enabling the trained robot policies to generalize across a combination of multiple visual disturbance types. Specifically, we introduce a multi-view representation learning approach fused with Spatial Transformer Network (STN) module to capture shared semantic information and correspondences among different viewpoints. In addition, we employ a curriculum-based randomization and augmentation approach to stabilize the RL training process and strengthen the visual generalization ability. To exhibit the effectiveness of Maniwhere, we meticulously design 8 tasks encompassing articulate objects, bi-manual, and dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual generalization and sim2real transfer abilities across 3 hardware platforms. Our experiments show that Maniwhere significantly outperforms existing state-of-the-art methods. Videos are provided at https://gemcollector.github.io/maniwhere/.