Advancing Neural Network Performance through Emergence-Promoting Initialization Scheme
This work addresses the challenge of improving neural network training efficiency and effectiveness for practitioners in machine learning, though it appears incremental as it builds on existing initialization methods like GradInit.
The paper tackled the problem of enhancing neural network performance by introducing an initialization scheme that promotes emergence, measured as structural nonlinearity, resulting in substantial improvements in model accuracy and training speed across various architectures.
Emergence in machine learning refers to the spontaneous appearance of complex behaviors or capabilities that arise from the scale and structure of training data and model architectures, despite not being explicitly programmed. We introduce a novel yet straightforward neural network initialization scheme that aims at achieving greater potential for emergence. Measuring emergence as a kind of structural nonlinearity, our method adjusts the layer-wise weight scaling factors to achieve higher emergence values. This enhancement is easy to implement, requiring no additional optimization steps for initialization compared to GradInit. We evaluate our approach across various architectures, including MLP and convolutional architectures for image recognition and transformers for machine translation. We demonstrate substantial improvements in both model accuracy and training speed, with and without batch normalization. The simplicity, theoretical innovation, and demonstrable empirical advantages of our method make it a potent enhancement to neural network initialization practices. These results suggest a promising direction for leveraging emergence to improve neural network training methodologies. Code is available at: https://github.com/johnnyjingzeli/EmergenceInit.