ConsistentDreamer: View-Consistent Meshes Through Balanced Multi-View Gaussian Optimization
This addresses the inconsistency issue in 3D generation for embodied AI simulations, representing an incremental improvement over prior diffusion-based methods.
The paper tackles the problem of inconsistent content and quality across views in image-to-3D generation by proposing ConsistentDreamer, which uses fixed multi-view prior images and dynamic weighting to balance rough shape and fine-detail optimization, resulting in better view consistency and visual quality compared to state-of-the-art methods.
Recent advances in diffusion models have significantly improved 3D generation, enabling the use of assets generated from an image for embodied AI simulations. However, the one-to-many nature of the image-to-3D problem limits their use due to inconsistent content and quality across views. Previous models optimize a 3D model by sampling views from a view-conditioned diffusion prior, but diffusion models cannot guarantee view consistency. Instead, we present ConsistentDreamer, where we first generate a set of fixed multi-view prior images and sample random views between them with another diffusion model through a score distillation sampling (SDS) loss. Thereby, we limit the discrepancies between the views guided by the SDS loss and ensure a consistent rough shape. In each iteration, we also use our generated multi-view prior images for fine-detail reconstruction. To balance between the rough shape and the fine-detail optimizations, we introduce dynamic task-dependent weights based on homoscedastic uncertainty, updated automatically in each iteration. Additionally, we employ opacity, depth distortion, and normal alignment losses to refine the surface for mesh extraction. Our method ensures better view consistency and visual quality compared to the state-of-the-art.