Neuron Campaign for Initialization Guided by Information Bottleneck Theory
This work addresses the initialization challenge for deep learning practitioners, offering an incremental improvement over existing methods by incorporating generalization insights.
The paper tackled the problem of improving generalization ability in deep neural network initialization by designing criteria based on Information Bottleneck theory and a neuron campaign algorithm, resulting in better generalization performance and faster convergence on the MNIST dataset.
Initialization plays a critical role in the training of deep neural networks (DNN). Existing initialization strategies mainly focus on stabilizing the training process to mitigate gradient vanish/explosion problems. However, these initialization methods are lacking in consideration about how to enhance generalization ability. The Information Bottleneck (IB) theory is a well-known understanding framework to provide an explanation about the generalization of DNN. Guided by the insights provided by IB theory, we design two criteria for better initializing DNN. And we further design a neuron campaign initialization algorithm to efficiently select a good initialization for a neural network on a given dataset. The experiments on MNIST dataset show that our method can lead to a better generalization performance with faster convergence.