MeshMamba: State Space Models for Articulated 3D Mesh Generation and Reconstruction
This addresses the problem of generating and reconstructing detailed 3D human meshes for applications in computer graphics and vision, representing an incremental advancement by applying a new model type to an existing domain.
The paper tackles 3D articulated mesh generation and reconstruction by introducing MeshMamba, which uses Mamba State Space Models to efficiently process over 10,000 vertices for capturing clothing and hand geometries. Results show MambaDiff3D outperforms previous approaches in 3D human shape generation, and Mamba-HMR extends whole-body reconstruction with competitive performance in near real-time.
In this paper, we introduce MeshMamba, a neural network model for learning 3D articulated mesh models by employing the recently proposed Mamba State Space Models (Mamba-SSMs). MeshMamba is efficient and scalable in handling a large number of input tokens, enabling the generation and reconstruction of body mesh models with more than 10,000 vertices, capturing clothing and hand geometries. The key to effectively learning MeshMamba is the serialization technique of mesh vertices into orderings that are easily processed by Mamba. This is achieved by sorting the vertices based on body part annotations or the 3D vertex locations of a template mesh, such that the ordering respects the structure of articulated shapes. Based on MeshMamba, we design 1) MambaDiff3D, a denoising diffusion model for generating 3D articulated meshes and 2) Mamba-HMR, a 3D human mesh recovery model that reconstructs a human body shape and pose from a single image. Experimental results showed that MambaDiff3D can generate dense 3D human meshes in clothes, with grasping hands, etc., and outperforms previous approaches in the 3D human shape generation task. Additionally, Mamba-HMR extends the capabilities of previous non-parametric human mesh recovery approaches, which were limited to handling body-only poses using around 500 vertex tokens, to the whole-body setting with face and hands, while achieving competitive performance in (near) real-time.