Towards a Mechanistic Explanation of Diffusion Model Generalization
This provides a mechanistic explanation for diffusion model generalization, which is an incremental but important step for understanding generative AI models.
The authors tackled the problem of explaining why diffusion models generalize well by identifying a shared local inductive bias across architectures and hypothesizing that network denoisers generalize through localized operations. They validated this by introducing novel denoising algorithms that replicate network behavior with lower mean squared error than previous methods.
We propose a simple, training-free mechanism which explains the generalization behaviour of diffusion models. By comparing pre-trained diffusion models to their theoretically optimal empirical counterparts, we identify a shared local inductive bias across a variety of network architectures. From this observation, we hypothesize that network denoisers generalize through localized denoising operations, as these operations approximate the training objective well over much of the training distribution. To validate our hypothesis, we introduce novel denoising algorithms which aggregate local empirical denoisers to replicate network behaviour. Comparing these algorithms to network denoisers across forward and reverse diffusion processes, our approach exhibits consistent visual similarity to neural network outputs, with lower mean squared error than previously proposed methods.