Information Potential Auto-Encoders
This work addresses the challenge of learning better features in auto-encoders for machine learning applications, but it is incremental as it builds on existing methods like VAEs.
The authors tackled the problem of improving representation learning in auto-encoders by proposing a framework that uses mutual information minimization as a regularization criterion, with experimental results showing that their non-parametric models offer more flexibility in handling complex distributions like Mixture of Gaussians compared to parametric methods like Variational Auto-Encoders.
In this paper, we suggest a framework to make use of mutual information as a regularization criterion to train Auto-Encoders (AEs). In the proposed framework, AEs are regularized by minimization of the mutual information between input and encoding variables of AEs during the training phase. In order to estimate the entropy of the encoding variables and the mutual information, we propose a non-parametric method. We also give an information theoretic view of Variational AEs (VAEs), which suggests that VAEs can be considered as parametric methods that estimate entropy. Experimental results show that the proposed non-parametric models have more degree of freedom in terms of representation learning of features drawn from complex distributions such as Mixture of Gaussians, compared to methods which estimate entropy using parametric approaches, such as Variational AEs.