ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections
This work addresses the challenge of 3D shape estimation from in-the-wild images for applications like animation and robotics, representing an incremental improvement over existing methods.
The paper tackles the problem of reconstructing 3D articulated shapes from noisy, sparse web images by proposing ARTIC3D, a self-supervised framework that uses 2D diffusion priors to enhance robustness and quality, resulting in outputs that are more robust to noise, higher in shape and texture details, and more realistic in animations.
Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging due to the ambiguities of camera viewpoint, pose, texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from a sparse image collection in-the-wild. Specifically, ARTIC3D is built upon a skeleton-based surface representation and is further guided by 2D diffusion priors from Stable Diffusion. First, we enhance the input images with occlusions/truncation via 2D diffusion to obtain cleaner mask estimates and semantic features. Second, we perform diffusion-guided 3D optimization to estimate shape and texture that are of high-fidelity and faithful to input images. We also propose a novel technique to calculate more stable image-level gradients via diffusion models compared to existing alternatives. Finally, we produce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations. Extensive evaluations on multiple existing datasets as well as newly introduced noisy web image collections with occlusions and truncation demonstrate that ARTIC3D outputs are more robust to noisy images, higher quality in terms of shape and texture details, and more realistic when animated. Project page: https://chhankyao.github.io/artic3d/