Limitations of Face Image Generation
This work addresses problems for researchers and practitioners using generated face images in data augmentation and model assessments, but it is incremental as it audits existing models without proposing new solutions.
The paper studied the limitations of text-to-image diffusion models in generating human faces, identifying issues such as faithfulness to prompts, demographic disparities, and distributional shifts through a framework combining qualitative and quantitative measures.
Text-to-image diffusion models have achieved widespread popularity due to their unprecedented image generation capability. In particular, their ability to synthesize and modify human faces has spurred research into using generated face images in both training data augmentation and model performance assessments. In this paper, we study the efficacy and shortcomings of generative models in the context of face generation. Utilizing a combination of qualitative and quantitative measures, including embedding-based metrics and user studies, we present a framework to audit the characteristics of generated faces conditioned on a set of social attributes. We applied our framework on faces generated through state-of-the-art text-to-image diffusion models. We identify several limitations of face image generation that include faithfulness to the text prompt, demographic disparities, and distributional shifts. Furthermore, we present an analytical model that provides insights into how training data selection contributes to the performance of generative models.