Analyzing and Improving the Skin Tone Consistency and Bias in Implicit 3D Relightable Face Generators
This addresses a fairness and bias issue in generative AI for face generation, particularly affecting individuals with dark skin tones, though it is incremental as it builds on existing implicit lighting methods.
The paper tackled the problem of skin tone inconsistency and bias in implicit 3D relightable face generators, where existing methods produce relit images with inconsistent skin tones and biased albedo towards lighter tones, particularly for dark skin under certain lighting conditions. The result was a proposed strategy that normalizes spherical harmonics coefficients and aligns them statistically, which effectively increased skin tone consistency and mitigated bias as demonstrated through experiments.
With the advances in generative adversarial networks (GANs) and neural rendering, 3D relightable face generation has received significant attention. Among the existing methods, a particularly successful technique uses an implicit lighting representation and generates relit images through the product of synthesized albedo and light-dependent shading images. While this approach produces high-quality results with intricate shading details, it often has difficulty producing relit images with consistent skin tones, particularly when the lighting condition is extracted from images of individuals with dark skin. Additionally, this technique is biased towards producing albedo images with lighter skin tones. Our main observation is that this problem is rooted in the biased spherical harmonics (SH) coefficients, used during training. Following this observation, we conduct an analysis and demonstrate that the bias appears not only in band 0 (DC term), but also in the other bands of the estimated SH coefficients. We then propose a simple, but effective, strategy to mitigate the problem. Specifically, we normalize the SH coefficients by their DC term to eliminate the inherent magnitude bias, while statistically align the coefficients in the other bands to alleviate the directional bias. We also propose a scaling strategy to match the distribution of illumination magnitude in the generated images with the training data. Through extensive experiments, we demonstrate the effectiveness of our solution in increasing the skin tone consistency and mitigating bias.