Analytically Tractable Inference in Deep Neural Networks
This provides an alternative training method for deep learning practitioners, though it is incremental as it builds on prior work for shallow networks.
The paper tackles the problem of training deep neural networks without relying on backpropagation by using the Tractable Approximate Gaussian Inference (TAGI) algorithm, showing that it matches or exceeds backpropagation's performance on classification tasks and generative adversarial networks with smaller architectures and fewer epochs.
Since its inception, deep learning has been overwhelmingly reliant on backpropagation and gradient-based optimization algorithms in order to learn weight and bias parameter values. Tractable Approximate Gaussian Inference (TAGI) algorithm was shown to be a viable and scalable alternative to backpropagation for shallow fully-connected neural networks. In this paper, we are demonstrating how TAGI matches or exceeds the performance of backpropagation, for training classic deep neural network architectures. Although TAGI's computational efficiency is still below that of deterministic approaches relying on backpropagation, it outperforms them on classification tasks and matches their performance for information maximizing generative adversarial networks while using smaller architectures trained with fewer epochs.