EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation
This work addresses the challenge of spatial perception for embodied AI systems, particularly in tabletop manipulation, though it appears incremental as it builds on existing masked autoencoder methods.
The paper tackles the problem of domain gaps and lack of 3D integration in robot manipulation by introducing EmbodiedMAE, a multi-modal masked autoencoder that learns from RGB, depth, and point cloud data, achieving state-of-the-art performance in 70 simulation and 20 real-world tasks.
We present EmbodiedMAE, a unified 3D multi-modal representation for robot manipulation. Current approaches suffer from significant domain gaps between training datasets and robot manipulation tasks, while also lacking model architectures that can effectively incorporate 3D information. To overcome these limitations, we enhance the DROID dataset with high-quality depth maps and point clouds, constructing DROID-3D as a valuable supplement for 3D embodied vision research. Then we develop EmbodiedMAE, a multi-modal masked autoencoder that simultaneously learns representations across RGB, depth, and point cloud modalities through stochastic masking and cross-modal fusion. Trained on DROID-3D, EmbodiedMAE consistently outperforms state-of-the-art vision foundation models (VFMs) in both training efficiency and final performance across 70 simulation tasks and 20 real-world robot manipulation tasks on two robot platforms. The model exhibits strong scaling behavior with size and promotes effective policy learning from 3D inputs. Experimental results establish EmbodiedMAE as a reliable unified 3D multi-modal VFM for embodied AI systems, particularly in precise tabletop manipulation settings where spatial perception is critical.