Reliability and Validity of Image-Based and Self-Reported Skin Phenotype Metrics
This addresses the need for objective skin-tone metrics in AI fairness evaluations, but it is incremental as it critiques existing methods without proposing a new solution.
The study tackled the problem of measuring skin-tone for evaluating face recognition system performance across demographic groups, finding that image-based estimates and self-reported Fitzpatrick Skin Types are unreliable, with significant variation relative to ground-truth device measurements, and that noisy estimates lead to errors in analyzing performance differentials.
With increasing adoption of face recognition systems, it is important to ensure adequate performance of these technologies across demographic groups. Recently, phenotypes such as skin-tone, have been proposed as superior alternatives to traditional race categories when exploring performance differentials. However, there is little consensus regarding how to appropriately measure skin-tone in evaluations of biometric performance or in AI more broadly. In this study, we explore the relationship between face-area-lightness-measures (FALMs) estimated from images and ground-truth skin readings collected using a device designed to measure human skin. FALMs estimated from different images of the same individual varied significantly relative to ground-truth FALM. This variation was only reduced by greater control of acquisition (camera, background, and environment). Next, we compare ground-truth FALM to Fitzpatrick Skin Types (FST) categories obtained using the standard, in-person, medical survey and show FST is poorly predictive of skin-tone. Finally, we show how noisy estimation of FALM leads to errors selecting explanatory factors for demographic differentials. These results demonstrate that measures of skin-tone for biometric performance evaluations must come from objective, characterized, and controlled sources. Further, despite this being a currently practiced approach, estimating FST categories and FALMs from uncontrolled imagery does not provide an appropriate measure of skin-tone.