Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales
For users of diffusion models who need both prompt alignment and group fairness, this work addresses a practical bottleneck where existing debiasing fails when guidance scale is adjusted.
Diffusion models' fairness degrades when users adjust the guidance scale, due to a previously overlooked 'guidance bias' that grows monotonically with scale. StayFair extends fairness across guidance scales by modifying only the guidance step, decoupling fairness from scale without sacrificing image quality.
Diffusion models steer conditional generation with a tunable guidance scale to trade off prompt alignment and diversity. However, existing debiasing techniques are optimized for a single scale, degrading fairness when users adjust this parameter. We trace this behavior to a previously overlooked source by decomposing total bias into two components: a model bias and a guidance bias. While prior work primarily targets the former, we show that the guidance bias grows monotonically with the guidance scale, eventually dominating the high-guidance regimes users prefer. To address this, we extend Strong Demographic Parity to guidance and derive a condition under which the target distribution retains its group ratio across guidance scales. We propose StayFair, which leverages this condition to design fair guidance algorithms in both regimes. For classifier guidance, it equalizes the classifier's output distributions across groups; for classifier-free guidance, it shifts the null embedding by a prompt-dependent offset. Because StayFair modifies only the guidance step, it is orthogonal to model debiasing and can be layered onto existing fair diffusion models to extend their fairness across guidance scales. Across class-conditional and text-to-image generation, StayFair decouples fairness from the guidance scale without sacrificing image quality.