Learning Coherent Matrixized Representation in Latent Space for Volumetric 4D Generation
This addresses the problem of limited motion diversity and continuity in 4D generation for applications in computer graphics and AI, representing an incremental advance over prior methods.
The paper tackles the challenge of generating volumetric 4D content with shape, color, and motion by proposing a framework that uses coherent 3D modeling and matrixized representation with spatio-temporal diffusion, resulting in high-quality 3D shapes and animations that improve over current methods on datasets like ShapeNet and Objaverse.
Directly learning to model 4D content, including shape, color, and motion, is challenging. Existing methods rely on pose priors for motion control, resulting in limited motion diversity and continuity in details. To address this, we propose a framework that generates volumetric 4D sequences, where 3D shapes are animated under given conditions (text-image guidance) with dynamic evolution in shape and color across spatial and temporal dimensions, allowing for free navigation and rendering from any direction. We first use a coherent 3D shape and color modeling to encode the shape and color of each detailed 3D geometry frame into a latent space. Then we propose a matrixized 4D sequence representation allowing efficient diffusion model operation. Finally, we introduce spatio-temporal diffusion for 4D volumetric generation under given images and text prompts. Extensive experiments on the ShapeNet, 3DBiCar, DeformingThings4D and Objaverse datasets for several tasks demonstrate that our method effectively learns to generate high quality 3D shapes with consistent color and coherent mesh animations, improving over the current methods. Our code will be publicly available.