Drill the Cork of Information Bottleneck by Inputting the Most Important Data
This work addresses the challenge of inefficient training in deep learning for researchers and practitioners, but it is incremental as it builds on existing information bottleneck theory and typicality sampling methods.
The paper tackles the problem of accelerating deep neural network training by using typicality sampling to boost the initial fitting phase described by the information bottleneck theory, showing that it increases the signal-to-noise ratio of gradient approximation and leads to faster training, with experimental results on synthetic and real-world datasets supporting these findings.
Deep learning has become the most powerful machine learning tool in the last decade. However, how to efficiently train deep neural networks remains to be thoroughly solved. The widely used minibatch stochastic gradient descent (SGD) still needs to be accelerated. As a promising tool to better understand the learning dynamic of minibatch SGD, the information bottleneck (IB) theory claims that the optimization process consists of an initial fitting phase and the following compression phase. Based on this principle, we further study typicality sampling, an efficient data selection method, and propose a new explanation of how it helps accelerate the training process of the deep networks. We show that the fitting phase depicted in the IB theory will be boosted with a high signal-to-noise ratio of gradient approximation if the typicality sampling is appropriately adopted. Furthermore, this finding also implies that the prior information of the training set is critical to the optimization process and the better use of the most important data can help the information flow through the bottleneck faster. Both theoretical analysis and experimental results on synthetic and real-world datasets demonstrate our conclusions.