When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models
This work addresses the problem of improving diffusion models for researchers and practitioners, but it is incremental as it explores an understudied application of ensembling without achieving consistent gains.
The paper investigated whether ensembling diffusion models improves generative modeling, finding that while it enhances score-matching loss and likelihood, it does not consistently improve perceptual quality metrics like FID on image datasets such as CIFAR-10 and FFHQ.
Diffusion models now generate high-quality, diverse samples, with an increasing focus on more powerful models. Although ensembling is a well-known way to improve supervised models, its application to unconditional score-based diffusion models remains largely unexplored. In this work we investigate whether it provides tangible benefits for generative modelling. We find that while ensembling the scores generally improves the score-matching loss and model likelihood, it fails to consistently enhance perceptual quality metrics such as FID on image datasets. We confirm this observation across a breadth of aggregation rules using Deep Ensembles, Monte Carlo Dropout, on CIFAR-10 and FFHQ. We attempt to explain this discrepancy by investigating possible explanations, such as the link between score estimation and image quality. We also look into tabular data through random forests, and find that one aggregation strategy outperforms the others. Finally, we provide theoretical insights into the summing of score models, which shed light not only on ensembling but also on several model composition techniques (e.g. guidance).