Merging of neural networks
This is an incremental improvement for machine learning practitioners seeking to optimize neural network training by avoiding unlucky initializations.
The paper tackles the problem of improving neural network performance by merging two networks trained with different initializations into a single network of the same size, showing that this approach leads to better performance than training a single network for longer.
We propose a simple scheme for merging two neural networks trained with different starting initialization into a single one with the same size as the original ones. We do this by carefully selecting channels from each input network. Our procedure might be used as a finalization step after one tries multiple starting seeds to avoid an unlucky one. We also show that training two networks and merging them leads to better performance than training a single network for an extended period of time. Availability: https://github.com/fmfi-compbio/neural-network-merging