MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers
This addresses the need for efficient mesh generation in computer graphics and 3D modeling, offering a novel method that improves over existing techniques, though it is incremental in applying transformer-based approaches to mesh data.
The paper tackles the problem of generating triangle meshes that are compact and artist-like, rather than dense from neural fields, by introducing MeshGPT, which uses decoder-only transformers to autoregressively generate meshes as sequences of triangles, resulting in a 9% increase in shape coverage and a 30-point improvement in FID scores.
We introduce MeshGPT, a new approach for generating triangle meshes that reflects the compactness typical of artist-created meshes, in contrast to dense triangle meshes extracted by iso-surfacing methods from neural fields. Inspired by recent advances in powerful large language models, we adopt a sequence-based approach to autoregressively generate triangle meshes as sequences of triangles. We first learn a vocabulary of latent quantized embeddings, using graph convolutions, which inform these embeddings of the local mesh geometry and topology. These embeddings are sequenced and decoded into triangles by a decoder, ensuring that they can effectively reconstruct the mesh. A transformer is then trained on this learned vocabulary to predict the index of the next embedding given previous embeddings. Once trained, our model can be autoregressively sampled to generate new triangle meshes, directly generating compact meshes with sharp edges, more closely imitating the efficient triangulation patterns of human-crafted meshes. MeshGPT demonstrates a notable improvement over state of the art mesh generation methods, with a 9% increase in shape coverage and a 30-point enhancement in FID scores across various categories.