Generalization Bound for Diffusion Models using Random Features
This work addresses the need for more interpretable and theoretically grounded generative models, though it appears incremental as it builds on existing random feature and diffusion model techniques.
The authors tackled the problem of diffusion models being computationally expensive and lacking theoretical justification by proposing a diffusion model-inspired deep random feature model that is interpretable and achieves comparable numerical results to a fully connected neural network with the same number of trainable parameters, validated on fashion MNIST and instrumental audio data.
Diffusion probabilistic models have been successfully used to generate data from noise. However, most diffusion models are computationally expensive and difficult to interpret with a lack of theoretical justification. Random feature models on the other hand have gained popularity due to their interpretability but their application to complex machine learning tasks remains limited. In this work, we present a diffusion model-inspired deep random feature model that is interpretable and gives comparable numerical results to a fully connected neural network having the same number of trainable parameters. Specifically, we extend existing results for random features and derive generalization bounds between the distribution of sampled data and the true distribution using properties of score matching. We validate our findings by generating samples on the fashion MNIST dataset and instrumental audio data.