DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
This addresses the problem of data scarcity for robotic manipulation by providing a unified representation that captures geometry, semantics, and dynamics, though it is incremental as it builds on existing self-supervised and 3D representation methods.
The paper tackles the challenge of learning generalizable robotic manipulation policies by introducing DynaRend, a framework that learns 3D-aware and dynamics-informed triplane features through masked reconstruction and future prediction, resulting in substantial improvements in policy success rates and generalization across benchmarks and real-world experiments.
Learning generalizable robotic manipulation policies remains a key challenge due to the scarcity of diverse real-world training data. While recent approaches have attempted to mitigate this through self-supervised representation learning, most either rely on 2D vision pretraining paradigms such as masked image modeling, which primarily focus on static semantics or scene geometry, or utilize large-scale video prediction models that emphasize 2D dynamics, thus failing to jointly learn the geometry, semantics, and dynamics required for effective manipulation. In this paper, we present DynaRend, a representation learning framework that learns 3D-aware and dynamics-informed triplane features via masked reconstruction and future prediction using differentiable volumetric rendering. By pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry, future dynamics, and task semantics in a unified triplane representation. The learned representations can be effectively transferred to downstream robotic manipulation tasks via action value map prediction. We evaluate DynaRend on two challenging benchmarks, RLBench and Colosseum, as well as in real-world robotic experiments, demonstrating substantial improvements in policy success rate, generalization to environmental perturbations, and real-world applicability across diverse manipulation tasks.