Learning 3D Dynamic Scene Representations for Robot Manipulation
This work addresses the challenge of robust 3D scene understanding for robotic manipulation, representing an incremental advance with specific applications.
The paper tackles the problem of representing 3D dynamic scenes for robot manipulation by introducing 3D Dynamic Scene Representation (DSR) and DSR-Net, which achieve state-of-the-art performance in modeling scene dynamics and enable accurate planning in tasks like planar pushing.
3D scene representation for robot manipulation should capture three key object properties: permanency -- objects that become occluded over time continue to exist; amodal completeness -- objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity -- the movement of each object is continuous over space and time. In this paper, we introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene representation that simultaneously discovers, tracks, reconstructs objects, and predicts their dynamics while capturing all three properties. We further propose DSR-Net, which learns to aggregate visual observations over multiple interactions to gradually build and refine DSR. Our model achieves state-of-the-art performance in modeling 3D scene dynamics with DSR on both simulated and real data. Combined with model predictive control, DSR-Net enables accurate planning in downstream robotic manipulation tasks such as planar pushing. Video is available at https://youtu.be/GQjYG3nQJ80.