Are Commercial Face Detection Models as Biased as Academic Models?
This research addresses fairness concerns in facial recognition systems for affected demographic groups, highlighting that commercial models may not mitigate biases despite greater resources.
The study compared academic and commercial face detection models for bias in noise robustness, finding that commercial models are always as biased or more biased than academic models, with significant performance disparities for older individuals and those presenting masculine gender expression.
As facial recognition systems are deployed more widely, scholars and activists have studied their biases and harms. Audits are commonly used to accomplish this and compare the algorithmic facial recognition systems' performance against datasets with various metadata labels about the subjects of the images. Seminal works have found discrepancies in performance by gender expression, age, perceived race, skin type, etc. These studies and audits often examine algorithms which fall into two categories: academic models or commercial models. We present a detailed comparison between academic and commercial face detection systems, specifically examining robustness to noise. We find that state-of-the-art academic face detection models exhibit demographic disparities in their noise robustness, specifically by having statistically significant decreased performance on older individuals and those who present their gender in a masculine manner. When we compare the size of these disparities to that of commercial models, we conclude that commercial models - in contrast to their relatively larger development budget and industry-level fairness commitments - are always as biased or more biased than an academic model.