Approximated Infomax Early Stopping: Revisiting Gaussian RBMs on Natural Images
This work addresses a specific issue in unsupervised learning for natural image representation, offering an incremental improvement for researchers in machine learning and computer vision.
The paper tackles the problem of Gaussian Restricted Boltzmann Machines (GRBMs) gaining non-overcomplete representations with uniform filters on natural images, proposing an early stopping technique based on approximated mutual information to achieve overcomplete representations that improve classifier performance.
We pursue an early stopping technique that helps Gaussian Restricted Boltzmann Machines (GRBMs) to gain good natural image representations in terms of overcompleteness and data fitting. GRBMs are widely considered as an unsuitable model for natural images because they gain non-overcomplete representations which include uniform filters that do not represent useful image features. We have recently found that GRBMs once gain and subsequently lose useful filters during their training, contrary to this common perspective. We attribute this phenomenon to a tradeoff between overcompleteness of GRBM representations and data fitting. To gain GRBM representations that are overcomplete and fit data well, we propose a measure for GRBM representation quality, approximated mutual information, and an early stopping technique based on this measure. The proposed method boosts performance of classifiers trained on GRBM representations.