Enhancing Diffusion Face Generation with Contrastive Embeddings and SegFormer Guidance
This work provides incremental improvements for researchers and practitioners in computer vision working on face generation with diffusion models in data-scarce scenarios.
The paper tackles the problem of controlled human face generation with diffusion models on limited data by integrating contrastive embedding learning and a SegFormer-based segmentation encoder, achieving improved semantic alignment and controllability in attribute-guided synthesis.
We present a benchmark of diffusion models for human face generation on a small-scale CelebAMask-HQ dataset, evaluating both unconditional and conditional pipelines. Our study compares UNet and DiT architectures for unconditional generation and explores LoRA-based fine-tuning of pretrained Stable Diffusion models as a separate experiment. Building on the multi-conditioning approach of Giambi and Lisanti, which uses both attribute vectors and segmentation masks, our main contribution is the integration of an InfoNCE loss for attribute embedding and the adoption of a SegFormer-based segmentation encoder. These enhancements improve the semantic alignment and controllability of attribute-guided synthesis. Our results highlight the effectiveness of contrastive embedding learning and advanced segmentation encoding for controlled face generation in limited data settings.