Phone2Proc: Bringing Robust Robots Into Our Chaotic World
This addresses the challenge of deploying robust robots in chaotic real-world settings like homes and offices, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the problem of sim-to-real generalization for embodied agents by introducing Phone2Proc, a method that uses a phone scan and procedural generation to create training scenes similar to target environments, resulting in an improvement from 34.7% to 70.7% success rate in ObjectNav performance across over 200 real-world trials.
Training embodied agents in simulation has become mainstream for the embodied AI community. However, these agents often struggle when deployed in the physical world due to their inability to generalize to real-world environments. In this paper, we present Phone2Proc, a method that uses a 10-minute phone scan and conditional procedural generation to create a distribution of training scenes that are semantically similar to the target environment. The generated scenes are conditioned on the wall layout and arrangement of large objects from the scan, while also sampling lighting, clutter, surface textures, and instances of smaller objects with randomized placement and materials. Leveraging just a simple RGB camera, training with Phone2Proc shows massive improvements from 34.7% to 70.7% success rate in sim-to-real ObjectNav performance across a test suite of over 200 trials in diverse real-world environments, including homes, offices, and RoboTHOR. Furthermore, Phone2Proc's diverse distribution of generated scenes makes agents remarkably robust to changes in the real world, such as human movement, object rearrangement, lighting changes, or clutter.