Towards understanding Diffusion Models (on Graphs)
This work provides incremental insights into diffusion models, aiming to enhance understanding for researchers in machine learning, particularly those interested in graph applications.
The paper investigates diffusion models by empirically examining the role of noise, sampling methods, and neural network complexity in a simpler setting to build foundational insights, with the long-term goal of applying these models to graphs.
Diffusion models have emerged from various theoretical and methodological perspectives, each offering unique insights into their underlying principles. In this work, we provide an overview of the most prominent approaches, drawing attention to their striking analogies -- namely, how seemingly diverse methodologies converge to a similar mathematical formulation of the core problem. While our ultimate goal is to understand these models in the context of graphs, we begin by conducting experiments in a simpler setting to build foundational insights. Through an empirical investigation of different diffusion and sampling techniques, we explore three critical questions: (1) What role does noise play in these models? (2) How significantly does the choice of the sampling method affect outcomes? (3) What function is the neural network approximating, and is high complexity necessary for optimal performance? Our findings aim to enhance the understanding of diffusion models and in the long run their application in graph machine learning.