SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance
This work addresses the need for high-quality, directly usable 3D avatars in graphics pipelines, offering a practical solution for applications like gaming or virtual reality, though it is incremental as it builds on existing text-to-image and 3D generation techniques.
The paper tackles the problem of generating photorealistic 3D avatars from text, which previous methods struggled with due to imprecise geometry and low-quality appearance, and achieves results that outperform prior methods by a large margin in both global and local geometry and appearance quality.
Powered by large-scale text-to-image generation models, text-to-3D avatar generation has made promising progress. However, most methods fail to produce photorealistic results, limited by imprecise geometry and low-quality appearance. Towards more practical avatar generation, we present SEEAvatar, a method for generating photorealistic 3D avatars from text with SElf-Evolving constraints for decoupled geometry and appearance. For geometry, we propose to constrain the optimized avatar in a decent global shape with a template avatar. The template avatar is initialized with human prior and can be updated by the optimized avatar periodically as an evolving template, which enables more flexible shape generation. Besides, the geometry is also constrained by the static human prior in local parts like face and hands to maintain the delicate structures. For appearance generation, we use diffusion model enhanced by prompt engineering to guide a physically based rendering pipeline to generate realistic textures. The lightness constraint is applied on the albedo texture to suppress incorrect lighting effect. Experiments show that our method outperforms previous methods on both global and local geometry and appearance quality by a large margin. Since our method can produce high-quality meshes and textures, such assets can be directly applied in classic graphics pipeline for realistic rendering under any lighting condition. Project page at: https://yoxu515.github.io/SEEAvatar/.