Neural network initialization with nonlinear characteristics and information on spectral bias
This work provides an incremental improvement in neural network initialization strategies for enhancing training performance.
The authors tackled the problem of neural network initialization by incorporating information about spectral bias, proposing a framework that adjusts scale factors in the SWIM algorithm to capture low-frequency components in early layers and high-frequency components in later layers. Numerical experiments on 1D regression and MNIST classification tasks showed that the proposed method outperforms conventional initialization algorithms.
Initialization of neural network parameters, such as weights and biases, has a crucial impact on learning performance; if chosen well, we can even avoid the need for additional training with backpropagation. For example, algorithms based on the ridgelet transform or the SWIM (sampling where it matters) concept have been proposed for initialization. On the other hand, it is well-known that neural networks tend to learn coarse information in the earlier layers. The feature is called spectral bias. In this work, we investigate the effects of utilizing information on the spectral bias in the initialization of neural networks. Hence, we propose a framework that adjusts the scale factors in the SWIM algorithm to capture low-frequency components in the early-stage hidden layers and to represent high-frequency components in the late-stage hidden layers. Numerical experiments on a one-dimensional regression task and the MNIST classification task demonstrate that the proposed method outperforms the conventional initialization algorithms. This work clarifies the importance of intrinsic spectral properties in learning neural networks, and the finding yields an effective parameter initialization strategy that enhances their training performance.