Saturating Auto-Encoders
This work addresses a specific issue in auto-encoder regularization for machine learning practitioners, but it is incremental as it builds on existing methods like Contractive and Sparse Auto-Encoders.
The authors tackled the problem of auto-encoders reconstructing inputs far from the data manifold by introducing a saturation regularizer that encourages activations in zero-gradient regions, resulting in improved feature learning and explicit limitation of reconstruction for non-manifold inputs.
We introduce a simple new regularizer for auto-encoders whose hidden-unit activation functions contain at least one zero-gradient (saturated) region. This regularizer explicitly encourages activations in the saturated region(s) of the corresponding activation function. We call these Saturating Auto-Encoders (SATAE). We show that the saturation regularizer explicitly limits the SATAE's ability to reconstruct inputs which are not near the data manifold. Furthermore, we show that a wide variety of features can be learned when different activation functions are used. Finally, connections are established with the Contractive and Sparse Auto-Encoders.