Improved Auto-Encoding using Deterministic Projected Belief Networks
This work addresses a problem for machine learning researchers and practitioners by proposing an incremental improvement to auto-encoders for better data representation.
The paper tackles the problem of improving auto-encoding performance by using deterministic projected belief networks (D-PBNs) with trainable compound activation functions (TCAs), resulting in significant outperformance over standard auto-encoders including variational auto-encoders.
In this paper, we exploit the unique properties of a deterministic projected belief network (D-PBN) to take full advantage of trainable compound activation functions (TCAs). A D-PBN is a type of auto-encoder that operates by "backing up" through a feed-forward neural network. TCAs are activation functions with complex monotonic-increasing shapes that change the distribution of the data so that the linear transformation that follows is more effective. Because a D-PBN operates by "backing up", the TCAs are inverted in the reconstruction process, restoring the original distribution of the data, thus taking advantage of a given TCA in both analysis and reconstruction. In this paper, we show that a D-PBN auto-encoder with TCAs can significantly out-perform standard auto-encoders including variational auto-encoders.