Statistical Guarantees for Approximate Stationary Points of Shallow Neural Networks
This work addresses the problem of aligning theoretical guarantees with practical neural network outputs for researchers and practitioners, though it is incremental as it is limited to shallow networks.
The paper tackles the gap between statistical theory and practice by developing statistical guarantees for stationary points of shallow neural networks, showing they match global optima up to logarithmic factors for linear networks and extending to ReLU networks under certain conditions.
Since statistical guarantees for neural networks are usually restricted to global optima of intricate objective functions, it is unclear whether these theories explain the performances of actual outputs of neural network pipelines. The goal of this paper is, therefore, to bring statistical theory closer to practice. We develop statistical guarantees for shallow linear neural networks that coincide up to logarithmic factors with the global optima but apply to stationary points and the points nearby. These results support the common notion that neural networks do not necessarily need to be optimized globally from a mathematical perspective. We then extend our statistical guarantees to shallow ReLU neural networks, assuming the first layer weight matrices are nearly identical for the stationary network and the target. More generally, despite being limited to shallow neural networks for now, our theories make an important step forward in describing the practical properties of neural networks in mathematical terms.