HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generation
This work improves text-to-3D generation for applications in graphics and AI by providing a novel alignment method, though it is incremental as it builds on existing CLIP-guided approaches.
The paper tackles the problem of text-to-3D generation by addressing the gap between text and image embeddings, proposing HOTS3D which uses spherical optimal transport to align features and achieves superior semantic consistency compared to state-of-the-art methods.
Recent CLIP-guided 3D generation methods have achieved promising results but struggle with generating faithful 3D shapes that conform with input text due to the gap between text and image embeddings. To this end, this paper proposes HOTS3D which makes the first attempt to effectively bridge this gap by aligning text features to the image features with spherical optimal transport(SOT). However, in high-dimensional situations, solving the SOT remains a challenge. To obtain the SOT map for high-dimensional features obtained from CLIP encoding of two modalities, we mathematically formulate and derive the solution based on Villani's theorem, which can directly align two hyper-sphere distributions without manifold exponential maps. Furthermore, we implement it by leveraging input convex neural networks (ICNNs) for the optimal Kantorovich potential. With the optimally mapped features, a diffusion-based generator is utilized to decode them into 3D shapes. Extensive quantitative and qualitative comparisons with state-of-the-art methods demonstrate the superiority of HOTS3D for text-to-3D generation, especially in the consistency with text semantics.