A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysis
This work addresses a specific technical bottleneck for researchers analyzing autoencoder features, offering an incremental improvement over existing dependence estimators.
The paper tackles the problem of measuring statistical dependence in deterministic autoencoders, which is ill-posed with traditional methods, by proposing a stable neural estimator based on a variational Gaussian formulation and orthonormal density-ratio decomposition, resulting in reduced computational cost and improved stability compared to MINE.
Statistical dependence measures like mutual information is ideal for analyzing autoencoders, but it can be ill-posed for deterministic, static, noise-free networks. We adopt the variational (Gaussian) formulation that makes dependence among inputs, latents, and reconstructions measurable, and we propose a stable neural dependence estimator based on an orthonormal density-ratio decomposition. Unlike MINE, our method avoids input concatenation and product-of-marginals re-pairing, reducing computational cost and improving stability. We introduce an efficient NMF-like scalar objective and demonstrate empirically that assuming Gaussian noise to form an auxiliary variable enables meaningful dependence measurements and supports quantitative feature analysis, with a sequential convergence of singular values.