Tighter Bounds on the Information Bottleneck with Application to Deep Learning
This work provides an incremental improvement to variational approximations of the Information Bottleneck, benefiting researchers and practitioners in deep learning by offering a simple method to boost adversarial robustness in classifier DNNs.
The paper tackles the intractability of the Information Bottleneck (IB) framework in deep learning by introducing a new, tighter variational bound, which improves the performance of IB-inspired deep neural networks and significantly enhances their adversarial robustness.
Deep Neural Nets (DNNs) learn latent representations induced by their downstream task, objective function, and other parameters. The quality of the learned representations impacts the DNN's generalization ability and the coherence of the emerging latent space. The Information Bottleneck (IB) provides a hypothetically optimal framework for data modeling, yet it is often intractable. Recent efforts combined DNNs with the IB by applying VAE-inspired variational methods to approximate bounds on mutual information, resulting in improved robustness to adversarial attacks. This work introduces a new and tighter variational bound for the IB, improving performance of previous IB-inspired DNNs. These advancements strengthen the case for the IB and its variational approximations as a data modeling framework, and provide a simple method to significantly enhance the adversarial robustness of classifier DNNs.