TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction
This work addresses the challenge of creating detailed 3D models for applications in computer graphics and AI, representing an incremental advancement by adapting next-token prediction to 3D generation.
The authors tackled the problem of generating high-quality 3D assets by proposing TAR3D, a framework that uses a 3D-aware VQ-VAE and a GPT for next-part prediction, achieving superior generation quality over existing methods in text-to-3D and image-to-3D tasks on ShapeNet and Objaverse datasets.
We present TAR3D, a novel framework that consists of a 3D-aware Vector Quantized-Variational AutoEncoder (VQ-VAE) and a Generative Pre-trained Transformer (GPT) to generate high-quality 3D assets. The core insight of this work is to migrate the multimodal unification and promising learning capabilities of the next-token prediction paradigm to conditional 3D object generation. To achieve this, the 3D VQ-VAE first encodes a wide range of 3D shapes into a compact triplane latent space and utilizes a set of discrete representations from a trainable codebook to reconstruct fine-grained geometries under the supervision of query point occupancy. Then, the 3D GPT, equipped with a custom triplane position embedding called TriPE, predicts the codebook index sequence with prefilling prompt tokens in an autoregressive manner so that the composition of 3D geometries can be modeled part by part. Extensive experiments on ShapeNet and Objaverse demonstrate that TAR3D can achieve superior generation quality over existing methods in text-to-3D and image-to-3D tasks