An information theoretic approach to the autoencoder
This work addresses representation learning for machine learning practitioners, offering an incremental improvement over existing autoencoder variants.
The authors tackled the problem of learning robust data representations by proposing an InfoMax Autoencoder (IMAE) that maximizes mutual information between input and hidden layers, resulting in strong clusterization performance on MNIST and Fashion-MNIST datasets.
We present a variation of the Autoencoder (AE) that explicitly maximizes the mutual information between the input data and the hidden representation. The proposed model, the InfoMax Autoencoder (IMAE), by construction is able to learn a robust representation and good prototypes of the data. IMAE is compared both theoretically and then computationally with the state of the art models: the Denoising and Contractive Autoencoders in the one-hidden layer setting and the Variational Autoencoder in the multi-layer case. Computational experiments are performed with the MNIST and Fashion-MNIST datasets and demonstrate particularly the strong clusterization performance of IMAE.