EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining
This addresses the need for 3D-aware models in egocentric video analysis, though it appears incremental by building on existing depth estimation and foundation models.
The paper tackles the problem of lacking 3D understanding in egocentric video-language pretraining by introducing EgoDTM, which incorporates 3D-aware learning from pseudo depth maps and enriched captions, achieving superior performance on diverse downstream tasks.
Egocentric video-language pretraining has significantly advanced video representation learning. Humans perceive and interact with a fully 3D world, developing spatial awareness that extends beyond text-based understanding. However, most previous works learn from 1D text or 2D visual cues, such as bounding boxes, which inherently lack 3D understanding. To bridge this gap, we introduce EgoDTM, an Egocentric Depth- and Text-aware Model, jointly trained through large-scale 3D-aware video pretraining and video-text contrastive learning. EgoDTM incorporates a lightweight 3D-aware decoder to efficiently learn 3D-awareness from pseudo depth maps generated by depth estimation models. To further facilitate 3D-aware video pretraining, we enrich the original brief captions with hand-object visual cues by organically combining several foundation models. Extensive experiments demonstrate EgoDTM's superior performance across diverse downstream tasks, highlighting its superior 3D-aware visual understanding. Our code will be released at https://github.com/xuboshen/EgoDTM.