DeFM: Learning Foundation Representations from Depth for Robotics
This work addresses the gap in depth-based foundation models for robotics, enabling robust, off-the-shelf representations for tasks like navigation and manipulation, though it is incremental as it adapts existing self-supervised methods to a new modality.
The authors tackled the underexplored problem of representation learning for depth images in robotics by developing DeFM, a self-supervised foundation model trained on 60M depth images, which achieves state-of-the-art performance on various benchmarks and demonstrates strong sim-to-real generalization.
Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/