From Pointwise to Powerhouse: Initialising Neural Networks with Generative Models
This work addresses the problem of improving neural network training efficiency and performance for researchers and practitioners, but it is incremental as it builds on existing generative model techniques.
The paper tackled the problem of traditional neural network initialization methods being limited to simple pointwise distributions by proposing new initialization methods using generative models, resulting in higher accuracy and faster initial convergence speed, though with diminished ensemble performance on out-of-distribution data that was addressed with a modification.
Traditional initialisation methods, e.g. He and Xavier, have been effective in avoiding the problem of vanishing or exploding gradients in neural networks. However, they only use simple pointwise distributions, which model one-dimensional variables. Moreover, they ignore most information about the architecture and disregard past training experiences. These limitations can be overcome by employing generative models for initialisation. In this paper, we introduce two groups of new initialisation methods. First, we locally initialise weight groups by employing variational autoencoders. Secondly, we globally initialise full weight sets by employing graph hypernetworks. We thoroughly evaluate the impact of the employed generative models on state-of-the-art neural networks in terms of accuracy, convergence speed and ensembling. Our results show that global initialisations result in higher accuracy and faster initial convergence speed. However, the implementation through graph hypernetworks leads to diminished ensemble performance on out of distribution data. To counteract, we propose a modification called noise graph hypernetwork, which encourages diversity in the produced ensemble members. Furthermore, our approach might be able to transfer learned knowledge to different image distributions. Our work provides insights into the potential, the trade-offs and possible modifications of these new initialisation methods.