Auto-encoders: reconstruction versus compression
This clarifies theoretical foundations for auto-encoder training, which is incremental for researchers in unsupervised learning.
The paper tackles the problem of distinguishing between training auto-encoders for reconstruction versus compression by showing that minimizing codelength is equivalent to minimizing reconstruction error plus corrective terms, which relate to denoising or contractive properties and determine an optimal noise level.
We discuss the similarities and differences between training an auto-encoder to minimize the reconstruction error, and training the same auto-encoder to compress the data via a generative model. Minimizing a codelength for the data using an auto-encoder is equivalent to minimizing the reconstruction error plus some correcting terms which have an interpretation as either a denoising or contractive property of the decoding function. These terms are related but not identical to those used in denoising or contractive auto-encoders [Vincent et al. 2010, Rifai et al. 2011]. In particular, the codelength viewpoint fully determines an optimal noise level for the denoising criterion.