Harnessing Diffusion-Generated Synthetic Images for Fair Image Classification
This addresses fairness issues in image classification for applications like face recognition, but it is incremental as it builds on existing diffusion and debiasing approaches.
The paper tackled bias in image classification due to uneven group representation by using diffusion-finetuning techniques like LoRA and DreamBooth to generate balanced synthetic training data, achieving results comparable to state-of-the-art debiasing methods and surpassing them as dataset bias severity increased.
Image classification systems often inherit biases from uneven group representation in training data. For example, in face datasets for hair color classification, blond hair may be disproportionately associated with females, reinforcing stereotypes. A recent approach leverages the Stable Diffusion model to generate balanced training data, but these models often struggle to preserve the original data distribution. In this work, we explore multiple diffusion-finetuning techniques, e.g., LoRA and DreamBooth, to generate images that more accurately represent each training group by learning directly from their samples. Additionally, in order to prevent a single DreamBooth model from being overwhelmed by excessive intra-group variations, we explore a technique of clustering images within each group and train a DreamBooth model per cluster. These models are then used to generate group-balanced data for pretraining, followed by fine-tuning on real data. Experiments on multiple benchmarks demonstrate that the studied finetuning approaches outperform vanilla Stable Diffusion on average and achieve results comparable to SOTA debiasing techniques like Group-DRO, while surpassing them as the dataset bias severity increases.