"Dependency Bottleneck" in Auto-encoding Architectures: an Empirical Study
This work addresses a methodological bottleneck for researchers studying generalization in deep neural networks, but it is incremental as it builds on existing information bottleneck frameworks.
The paper tackles the issue of inaccurate mutual information measurement in deep neural networks due to density estimation by proposing the use of Hilbert-Schmidt Independence Criterion (HSIC) as a dependency measure, and empirically evaluates its generalization properties in auto-encoding architectures.
Recent works investigated the generalization properties in deep neural networks (DNNs) by studying the Information Bottleneck in DNNs. However, the mea- surement of the mutual information (MI) is often inaccurate due to the density estimation. To address this issue, we propose to measure the dependency instead of MI between layers in DNNs. Specifically, we propose to use Hilbert-Schmidt Independence Criterion (HSIC) as the dependency measure, which can measure the dependence of two random variables without estimating probability densities. Moreover, HSIC is a special case of the Squared-loss Mutual Information (SMI). In the experiment, we empirically evaluate the generalization property using HSIC in both the reconstruction and prediction auto-encoding (AE) architectures.