Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum
This provides a theoretical insight into local minima characterization and inductive bias in neural networks, though it is incremental as it builds on existing ridgelet transform frameworks.
The study tackled the problem of characterizing local minima in over-parametrized neural networks by analyzing parameter distributions under ridge-regularized empirical risk minimization, showing that these distributions converge to the ridgelet transform spectrum, as confirmed through numerical experiments.
Characterization of local minima draws much attention in theoretical studies of deep learning. In this study, we investigate the distribution of parameters in an over-parametrized finite neural network trained by ridge regularized empirical square risk minimization (RERM). We develop a new theory of ridgelet transform, a wavelet-like integral transform that provides a powerful and general framework for the theoretical study of neural networks involving not only the ReLU but general activation functions. We show that the distribution of the parameters converges to a spectrum of the ridgelet transform. This result provides a new insight into the characterization of the local minima of neural networks, and the theoretical background of an inductive bias theory based on lazy regimes. We confirm the visual resemblance between the parameter distribution trained by SGD, and the ridgelet spectrum calculated by numerical integration through numerical experiments with finite models.