Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks
For researchers in deep learning theory, this provides a principled variational framework that bridges NTK and feature-learning regimes, offering a new perspective on over-parameterization.
This work replaces the discrete training of shallow neural networks with a well-posed continuum variational surrogate, proving that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization. They establish generalization error controls at a rate of 1/α and show that finite-width networks achieve the continuum optimum at an O(1/N) rate.
Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics. In this work, we propose a paradigm shift by replacing the discrete training problem of shallow neural networks with a well-posed continuum variational surrogate. We identify a family of $λ$-convex functionals over parameter densities in weighted Sobolev spaces and prove that these variational problems are globally well-posed, stable, and exhibit unexpected almost $C^3$ regularity. Unlike existing Wasserstein-based or Mean-Field approaches, which often face limited regularity and discretization challenges, our formulation provides direct access to elliptic regularity and convex analysis. This allows us to prove that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization entirely. We establish explicit generalization error controls at a rate of $1/α$ relative to the regularization parameter, and prove that finite-width networks of size $N$ achieve the continuum optimum at an $O(1/N)$ rate. This perspective bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, providing a principled framework for understanding over-parameterization through the lens of variational calculus.