Representing 3D Shapes With 64 Latent Vectors for 3D Diffusion Models
This addresses the computational bottleneck in 3D diffusion models for applications like 3D content creation, though it is an incremental improvement on existing VAE-based compression methods.
The paper tackles the problem of inefficient 3D diffusion models by introducing COD-VAE, which compresses 3D shapes into 64 latent vectors, achieving 16x compression while maintaining quality and enabling a 20.8x speedup in generation.
Constructing a compressed latent space through a variational autoencoder (VAE) is the key for efficient 3D diffusion models. This paper introduces COD-VAE that encodes 3D shapes into a COmpact set of 1D latent vectors without sacrificing quality. COD-VAE introduces a two-stage autoencoder scheme to improve compression and decoding efficiency. First, our encoder block progressively compresses point clouds into compact latent vectors via intermediate point patches. Second, our triplane-based decoder reconstructs dense triplanes from latent vectors instead of directly decoding neural fields, significantly reducing computational overhead of neural fields decoding. Finally, we propose uncertainty-guided token pruning, which allocates resources adaptively by skipping computations in simpler regions and improves the decoder efficiency. Experimental results demonstrate that COD-VAE achieves 16x compression compared to the baseline while maintaining quality. This enables 20.8x speedup in generation, highlighting that a large number of latent vectors is not a prerequisite for high-quality reconstruction and generation. The code is available at https://github.com/join16/COD-VAE.