Generalized Information Bottleneck for Gaussian Variables
This work provides a theoretical advance for researchers in representation learning, offering a strategy to approximate the original IB problem, though it is incremental as it builds on existing IB frameworks.
The authors tackled the computational intractability of the information bottleneck (IB) method by generalizing it with Renyi and Jeffreys divergences for Gaussian variables, deriving an exact analytical solution that reveals structural transitions and shows these alternative measures perform well under the original IB objective.
The information bottleneck (IB) method offers an attractive framework for understanding representation learning, however its applications are often limited by its computational intractability. Analytical characterization of the IB method is not only of practical interest, but it can also lead to new insights into learning phenomena. Here we consider a generalized IB problem, in which the mutual information in the original IB method is replaced by correlation measures based on Renyi and Jeffreys divergences. We derive an exact analytical IB solution for the case of Gaussian correlated variables. Our analysis reveals a series of structural transitions, similar to those previously observed in the original IB case. We find further that although solving the original, Renyi and Jeffreys IB problems yields different representations in general, the structural transitions occur at the same critical tradeoff parameters, and the Renyi and Jeffreys IB solutions perform well under the original IB objective. Our results suggest that formulating the IB method with alternative correlation measures could offer a strategy for obtaining an approximate solution to the original IB problem.