Visualizing Information Bottleneck through Variational Inference
This work provides incremental insights into training dynamics for researchers in deep learning theory.
The authors tackled the problem of analyzing SGD training phases in deep neural networks on MNIST classification, confirming the existence of fitting and compression phases as predicted by Information Bottleneck theory, and proposed a variational inference method for estimating mutual information.
The Information Bottleneck theory provides a theoretical and computational framework for finding approximate minimum sufficient statistics. Analysis of the Stochastic Gradient Descent (SGD) training of a neural network on a toy problem has shown the existence of two phases, fitting and compression. In this work, we analyze the SGD training process of a Deep Neural Network on MNIST classification and confirm the existence of two phases of SGD training. We also propose a setup for estimating the mutual information for a Deep Neural Network through Variational Inference.