The Variational InfoMax Learning Objective
This work addresses a theoretical and practical problem for machine learning researchers by providing a more direct optimization method for popular objectives, though it appears incremental as it builds on existing variational frameworks.
The paper tackles the equivalence of Bayesian Inference and Information Bottleneck objectives to InfoMax, and introduces Variational InfoMax (VIM) to directly optimize them, resulting in improved accuracy, robustness to noise, and representation quality over Variational Information Bottleneck in computational experiments.
Bayesian Inference and Information Bottleneck are the two most popular objectives for neural networks, but they can be optimised only via a variational lower bound: the Variational Information Bottleneck (VIB). In this manuscript we show that the two objectives are actually equivalent to the InfoMax: maximise the information between the data and the labels. The InfoMax representation of the two objectives is not relevant only per se, since it helps to understand the role of the network capacity, but also because it allows us to derive a variational objective, the Variational InfoMax (VIM), that maximises them directly without resorting to any lower bound. The theoretical improvement of VIM over VIB is highlighted by the computational experiments, where the model trained by VIM improves the VIB model in three different tasks: accuracy, robustness to noise and representation quality.