PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation
This work simplifies monocular 3D reconstruction by showing that a plain ViT-based diffusion model can outperform complex hybrid and latent-based approaches, benefiting researchers seeking efficient and effective geometry estimation.
The paper introduces PointDiT, a minimalist pixel-space Diffusion Transformer for monocular geometry estimation that operates directly on raw 3D point map patches, achieving state-of-the-art results without complex hybrid architectures or latent compression. It surpasses latent-based diffusion models while being simpler, producing sharper geometry and better robustness in ambiguous regions.
State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.