3DILG: Irregular Latent Grids for 3D Generative Modeling
This work addresses the problem of efficient and adaptive 3D shape encoding for computer vision and graphics researchers, offering incremental improvements over existing grid-based methods.
The paper tackles 3D shape representation by proposing irregular latent grids for neural fields, improving reconstruction accuracy from point clouds and enabling high-quality shape generation with auto-regressive models, achieving state-of-the-art results in generative 3D modeling.
We propose a new representation for encoding 3D shapes as neural fields. The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation. Existing works on neural fields are grid-based representations with latents defined on a regular grid. In contrast, we define latents on irregular grids, enabling our representation to be sparse and adaptive. In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid-based methods in terms of reconstruction accuracy. For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models. We show different applications that improve over the current state of the art. First, we show results for probabilistic shape reconstruction from a single higher resolution image. Second, we train a probabilistic model conditioned on very low resolution images. Third, we apply our model to category-conditioned generation. All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.