MeTTA: Single-View to 3D Textured Mesh Reconstruction with Test-Time Adaptation
This addresses the problem of 3D reconstruction from single images for computer vision and graphics applications, but it is incremental as it builds on existing learning-based methods with test-time adaptation.
The paper tackles the challenge of reconstructing 3D textured meshes from single-view images, especially for out-of-distribution cases, by proposing MeTTA, a test-time adaptation method that uses generative priors and joint optimization to handle unseen samples, resulting in effective adaptation and realistic appearance with PBR textures.
Reconstructing 3D from a single view image is a long-standing challenge. One of the popular approaches to tackle this problem is learning-based methods, but dealing with the test cases unfamiliar with training data (Out-of-distribution; OoD) introduces an additional challenge. To adapt for unseen samples in test time, we propose MeTTA, a test-time adaptation (TTA) exploiting generative prior. We design joint optimization of 3D geometry, appearance, and pose to handle OoD cases with only a single view image. However, the alignment between the reference image and the 3D shape via the estimated viewpoint could be erroneous, which leads to ambiguity. To address this ambiguity, we carefully design learnable virtual cameras and their self-calibration. In our experiments, we demonstrate that MeTTA effectively deals with OoD scenarios at failure cases of existing learning-based 3D reconstruction models and enables obtaining a realistic appearance with physically based rendering (PBR) textures.