Variance-Aware Adaptive Weighting for Diffusion Model Training
This work addresses a specific bottleneck in diffusion model training for generative modeling, offering an incremental improvement to stabilize and enhance optimization.
The paper tackles the problem of imbalanced training dynamics across noise levels in diffusion models, which leads to inefficient optimization and unstable learning, by proposing a variance-aware adaptive weighting strategy that improves generative performance, achieving lower Fréchet Inception Distance (FID) and reducing performance variance across random seeds on CIFAR-10 and CIFAR-100.
Diffusion models have recently achieved remarkable success in generative modeling, yet their training dynamics across different noise levels remain highly imbalanced, which can lead to inefficient optimization and unstable learning behavior. In this work, we investigate this imbalance from the perspective of loss variance across log-SNR levels and propose a variance-aware adaptive weighting strategy to address it. The proposed approach dynamically adjusts training weights based on the observed variance distribution, encouraging a more balanced optimization process across noise levels. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed method consistently improves generative performance over standard training schemes, achieving lower Fréchet Inception Distance (FID) while also reducing performance variance across random seeds. Additional analysis, including loss-log-SNR visualization, variance heatmaps, and ablation studies, further reveal that the adaptive weighting effectively stabilizes training dynamics. These results highlight the potential of variance-aware training strategies for improving diffusion model optimization.