Learning Robust Variational Information Bottleneck with Reference
This work addresses robustness in machine learning models for classification tasks, but it is incremental as it builds on existing VIB methods with specific refinements.
The paper tackles the problem of improving robustness to adversarial perturbations in variational information bottleneck (VIB) training by refining categorical class information with soft labels from a pre-trained reference network and relaxing the Gaussian posterior assumption. Results on MNIST and CIFAR-10 datasets show significant outperformance over benchmarked models.
We propose a new approach to train a variational information bottleneck (VIB) that improves its robustness to adversarial perturbations. Unlike the traditional methods where the hard labels are usually used for the classification task, we refine the categorical class information in the training phase with soft labels which are obtained from a pre-trained reference neural network and can reflect the likelihood of the original class labels. We also relax the Gaussian posterior assumption in the VIB implementation by using the mutual information neural estimation. Extensive experiments have been performed with the MNIST and CIFAR-10 datasets, and the results show that our proposed approach significantly outperforms the benchmarked models.