GaussianAnything: Interactive Point Cloud Flow Matching For 3D Object Generation
This work addresses problems in 3D object generation for content creators, offering interactive editing capabilities, but it appears incremental as it builds on existing methods like VAEs and flow matching.
The paper tackles challenges in 3D content generation by introducing a framework that uses a VAE with multi-view renderings and a cascaded latent flow-based model, achieving scalable, high-quality 3D generation with geometry-texture disentanglement and outperforming existing methods in text- and image-conditioned tasks.
While 3D content generation has advanced significantly, existing methods still face challenges with input formats, latent space design, and output representations. This paper introduces a novel 3D generation framework that addresses these challenges, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder (VAE) with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information, and incorporates a cascaded latent flow-based model for improved shape-texture disentanglement. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single image inputs. Notably, the newly proposed latent space naturally enables geometry-texture disentanglement, thus allowing 3D-aware editing. Experimental results demonstrate the effectiveness of our approach on multiple datasets, outperforming existing native 3D methods in both text- and image-conditioned 3D generation.