Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
This work addresses a theoretical bottleneck for researchers and practitioners in deep learning by making information-theoretic training objectives more practical, though it is incremental as it builds on existing principles.
The paper tackles the challenge of applying the Information Bottleneck principle to train deep neural networks by unifying competing objectives and developing surrogate objectives that avoid cumbersome computations, demonstrating this on datasets like MNIST, CIFAR-10, and Imagenette with ResNet architectures.
The Information Bottleneck principle offers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models. However, multiple competing objectives are proposed in the literature, and the information-theoretic quantities used in these objectives are difficult to compute for large deep neural networks, which in turn limits their use as a training objective. In this work, we review these quantities and compare and unify previously proposed objectives, which allows us to develop surrogate objectives more friendly to optimization without relying on cumbersome tools such as density estimation. We find that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures. We demonstrate our insights on MNIST, CIFAR-10 and Imagenette with modern DNN architectures (ResNets).