Synthetic Counterfactual Faces
This work addresses the need for annotated data to test computer vision systems in biometric applications, though it is incremental as it builds on existing generative methods for synthetic data creation.
The authors tackled the problem of evaluating robustness and fairness of computer vision systems for faces by developing a generative AI framework to create targeted, high-quality synthetic counterfactual face data, which they validated through user studies and used to identify facial attributes that cause failures in a leading commercial vision model.
Computer vision systems have been deployed in various applications involving biometrics like human faces. These systems can identify social media users, search for missing persons, and verify identity of individuals. While computer vision models are often evaluated for accuracy on available benchmarks, more annotated data is necessary to learn about their robustness and fairness against semantic distributional shifts in input data, especially in face data. Among annotated data, counterfactual examples grant strong explainability characteristics. Because collecting natural face data is prohibitively expensive, we put forth a generative AI-based framework to construct targeted, counterfactual, high-quality synthetic face data. Our synthetic data pipeline has many use cases, including face recognition systems sensitivity evaluations and image understanding system probes. The pipeline is validated with multiple user studies. We showcase the efficacy of our face generation pipeline on a leading commercial vision model. We identify facial attributes that cause vision systems to fail.