Relevant sparse codes with variational information bottleneck
This work addresses the computational bottleneck in extracting relevant data aspects for applications requiring sparse feature selection, though it is incremental as it builds on existing IB methods.
The authors tackled the computational challenge of the information bottleneck (IB) method for high-dimensional or non-Gaussian data by proposing a variational approximation scheme, resulting in an algorithm that recovers relevant and sparse features and extends to non-linear problems via kernelization.
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunately however, the IB method is computationally demanding when data are high-dimensional and/or non-gaussian. Here we propose an approximate variational scheme for maximizing a lower bound on the IB objective, analogous to variational EM. Using this method, we derive an IB algorithm to recover features that are both relevant and sparse. Finally, we demonstrate how kernelized versions of the algorithm can be used to address a broad range of problems with non-linear relation between X and Y.