HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D
This work addresses a key challenge in 3D content creation from 2D images, offering a method to enhance novel-view diversity while maintaining multi-view coherency, though it appears incremental in the context of existing diffusion-based approaches.
The paper tackles the problem of balancing consistency and diversity in single-image 3D generation, introducing HarmonyView, a diffusion sampling technique that achieves a win-win scenario in both aspects.
Recent progress in single-image 3D generation highlights the importance of multi-view coherency, leveraging 3D priors from large-scale diffusion models pretrained on Internet-scale images. However, the aspect of novel-view diversity remains underexplored within the research landscape due to the ambiguity in converting a 2D image into 3D content, where numerous potential shapes can emerge. Here, we aim to address this research gap by simultaneously addressing both consistency and diversity. Yet, striking a balance between these two aspects poses a considerable challenge due to their inherent trade-offs. This work introduces HarmonyView, a simple yet effective diffusion sampling technique adept at decomposing two intricate aspects in single-image 3D generation: consistency and diversity. This approach paves the way for a more nuanced exploration of the two critical dimensions within the sampling process. Moreover, we propose a new evaluation metric based on CLIP image and text encoders to comprehensively assess the diversity of the generated views, which closely aligns with human evaluators' judgments. In experiments, HarmonyView achieves a harmonious balance, demonstrating a win-win scenario in both consistency and diversity.