Gradient Descent as a Shrinkage Operator for Spectral Bias
This provides theoretical insights into spectral bias control for researchers studying neural network optimization and generalization.
The authors characterized how activation functions influence spectral bias in shallow networks and reinterpreted gradient descent as a shrinkage operator that controls spectral bias by masking Jacobian singular values. They showed GD regularization only works with monotonic activations and proposed non-monotonic activations as iteration-efficient alternatives.
We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. We then demonstrate how gradient descent (GD) can be reinterpreted as a shrinkage operator that masks the singular values of a neural network's Jacobian. Viewed this way, GD implicitly selects the number of frequency components to retain, thereby controlling the spectral bias. An explicit relationship is proposed between the choice of GD hyperparameters (learning rate & number of iterations) and bandwidth (the number of active components). GD regularization is shown to be effective only with monotonic activation functions. Finally, we highlight the utility of non-monotonic activation functions (sinc, Gaussian) as iteration-efficient surrogates for spectral bias.