Bringing Diversity from Diffusion Models to Semantic-Guided Face Asset Generation
This work addresses the challenge of generating diverse and expressive digital face assets for applications like modeling and reconstruction, though it is incremental as it builds on existing diffusion and GAN methods.
The paper tackled the problem of limited diversity and control in digital face modeling by introducing a semantically controllable generative network that uses a diffusion model to create a high-quality 3D face database of 44,000 models, enabling enhanced control and continuous editing of facial attributes.
Digital modeling and reconstruction of human faces serve various applications. However, its availability is often hindered by the requirements of data capturing devices, manual labor, and suitable actors. This situation restricts the diversity, expressiveness, and control over the resulting models. This work aims to demonstrate that a semantically controllable generative network can provide enhanced control over the digital face modeling process. To enhance diversity beyond the limited human faces scanned in a controlled setting, we introduce a novel data generation pipeline that creates a high-quality 3D face database using a pre-trained diffusion model. Our proposed normalization module converts synthesized data from the diffusion model into high-quality scanned data. Using the 44,000 face models we obtained, we further developed an efficient GAN-based generator. This generator accepts semantic attributes as input, and generates geometry and albedo. It also allows continuous post-editing of attributes in the latent space. Our asset refinement component subsequently creates physically-based facial assets. We introduce a comprehensive system designed for creating and editing high-quality face assets. Our proposed model has undergone extensive experiment, comparison and evaluation. We also integrate everything into a web-based interactive tool. We aim to make this tool publicly available with the release of the paper.