Finite Sample Identification of Wide Shallow Neural Networks with Biases
This work addresses a gap in understanding parameter identification for neural networks with biases, which is important for researchers in machine learning theory and optimization, though it is incremental as it builds on prior work in the teacher-student model.
The paper tackles the finite sample identification problem for wide shallow neural networks with biases, providing constructive methods and theoretical guarantees for networks with a number of neurons between D and D^2, where D is the input dimension, and demonstrates effectiveness through numerical results.
Artificial neural networks are functions depending on a finite number of parameters typically encoded as weights and biases. The identification of the parameters of the network from finite samples of input-output pairs is often referred to as the \emph{teacher-student model}, and this model has represented a popular framework for understanding training and generalization. Even if the problem is NP-complete in the worst case, a rapidly growing literature -- after adding suitable distributional assumptions -- has established finite sample identification of two-layer networks with a number of neurons $m=\mathcal O(D)$, $D$ being the input dimension. For the range $D<m<D^2$ the problem becomes harder, and truly little is known for networks parametrized by biases as well. This paper fills the gap by providing constructive methods and theoretical guarantees of finite sample identification for such wider shallow networks with biases. Our approach is based on a two-step pipeline: first, we recover the direction of the weights, by exploiting second order information; next, we identify the signs by suitable algebraic evaluations, and we recover the biases by empirical risk minimization via gradient descent. Numerical results demonstrate the effectiveness of our approach.