SemUV: Deep Learning based semantic manipulation over UV texture map of virtual human heads
This work addresses the need for enhanced control and precision in appearance manipulation for graphic designers in AR, VR, gaming, and VFX, though it is incremental as it builds on existing StyleGAN methods applied to a new domain.
The paper tackles the problem of semantic manipulation of virtual human heads in 3D applications by introducing SemUV, a deep learning approach that operates directly in the UV texture space, demonstrating superior identity preservation and effective modification of features like age, gender, and facial hair compared to 2D techniques.
Designing and manipulating virtual human heads is essential across various applications, including AR, VR, gaming, human-computer interaction and VFX. Traditional graphic-based approaches require manual effort and resources to achieve accurate representation of human heads. While modern deep learning techniques can generate and edit highly photorealistic images of faces, their focus remains predominantly on 2D facial images. This limitation makes them less suitable for 3D applications. Recognizing the vital role of editing within the UV texture space as a key component in the 3D graphics pipeline, our work focuses on this aspect to benefit graphic designers by providing enhanced control and precision in appearance manipulation. Research on existing methods within the UV texture space is limited, complex, and poses challenges. In this paper, we introduce SemUV: a simple and effective approach using the FFHQ-UV dataset for semantic manipulation directly within the UV texture space. We train a StyleGAN model on the publicly available FFHQ-UV dataset, and subsequently train a boundary for interpolation and semantic feature manipulation. Through experiments comparing our method with 2D manipulation technique, we demonstrate its superior ability to preserve identity while effectively modifying semantic features such as age, gender, and facial hair. Our approach is simple, agnostic to other 3D components such as structure, lighting, and rendering, and also enables seamless integration into standard 3D graphics pipelines without demanding extensive domain expertise, time, or resources.