Towards universal neural nets: Gibbs machines and ACE
This work proposes a new framework for incremental learning in neural networks, potentially benefiting researchers in machine learning and physics.
The authors tackled the problem of creating universal generative neural networks by introducing Gibbs machines and stochastic auto-classifier-encoders (ACE), achieving state-of-the-art performance in classification and density estimation on the MNIST dataset.
We study from a physics viewpoint a class of generative neural nets, Gibbs machines, designed for gradual learning. While including variational auto-encoders, they offer a broader universal platform for incrementally adding newly learned features, including physical symmetries. Their direct connection to statistical physics and information geometry is established. A variational Pythagorean theorem justifies invoking the exponential/Gibbs class of probabilities for creating brand new objects. Combining these nets with classifiers, gives rise to a brand of universal generative neural nets - stochastic auto-classifier-encoders (ACE). ACE have state-of-the-art performance in their class, both for classification and density estimation for the MNIST data set.