Gaussian Universality in Neural Network Dynamics with Generalized Structured Input Distributions
This provides incremental theoretical support for understanding deep learning models by bridging practical performance with foundational theory.
The paper tackled the problem of analyzing neural network dynamics under more realistic structured input distributions, such as Gaussian mixtures, and found that with standardization, the model behavior converges to that under Gaussian assumptions, demonstrating universality across diverse distributions.
Bridging the gap between the practical performance of deep learning and its theoretical foundations often involves analyzing neural networks through stochastic gradient descent (SGD). Expanding on previous research that focused on modeling structured inputs under a simple Gaussian setting, we analyze the behavior of a deep learning system trained on inputs modeled as Gaussian mixtures to better simulate more general structured inputs. Through empirical analysis and theoretical investigation, we demonstrate that under certain standardization schemes, the deep learning model converges toward Gaussian setting behavior, even when the input data follow more complex or real-world distributions. This finding exhibits a form of universality in which diverse structured distributions yield results consistent with Gaussian assumptions, which can support the theoretical understanding of deep learning models.