A spectral-based analysis of the separation between two-layer neural networks and linear methods
This provides theoretical insights into the advantages of neural networks over linear methods, which is incremental but clarifies activation function effects for researchers in machine learning theory.
The paper tackles the problem of quantifying the separation between two-layer neural networks and linear methods in approximating high-dimensional functions, showing that this can be reduced to estimating Kolmogorov widths and characterized via kernel spectra, with results including sharper bounds for nonsmooth activations and conditions for negligible separation with smooth activations.
We propose a spectral-based approach to analyze how two-layer neural networks separate from linear methods in terms of approximating high-dimensional functions. We show that quantifying this separation can be reduced to estimating the Kolmogorov width of two-layer neural networks, and the latter can be further characterized by using the spectrum of an associated kernel. Different from previous work, our approach allows obtaining upper bounds, lower bounds, and identifying explicit hard functions in a united manner. We provide a systematic study of how the choice of activation functions affects the separation, in particular the dependence on the input dimension. Specifically, for nonsmooth activation functions, we extend known results to more activation functions with sharper bounds. As concrete examples, we prove that any single neuron can instantiate the separation between neural networks and random feature models. For smooth activation functions, one surprising finding is that the separation is negligible unless the norms of inner-layer weights are polynomially large with respect to the input dimension. By contrast, the separation for nonsmooth activation functions is independent of the norms of inner-layer weights.