Analytically Tractable Hidden-States Inference in Bayesian Neural Networks
This work addresses the challenge of enabling analytically tractable inference for hidden states in neural networks, which could benefit researchers and practitioners in machine learning by providing an alternative to gradient-based methods, though it appears incremental as it builds on existing TAGI capabilities.
The paper tackles the problem of performing hidden-states inference in Bayesian neural networks, which was previously considered intractable, by leveraging tractable approximate Gaussian inference (TAGI) to infer hidden states with constraints for specific objectives, resulting in applications such as generating adversarial-attack examples, using neural networks for black-box optimization, and applying inference in continuous-action reinforcement learning.
With few exceptions, neural networks have been relying on backpropagation and gradient descent as the inference engine in order to learn the model parameters, because the closed-form Bayesian inference for neural networks has been considered to be intractable. In this paper, we show how we can leverage the tractable approximate Gaussian inference's (TAGI) capabilities to infer hidden states, rather than only using it for inferring the network's parameters. One novel aspect it allows is to infer hidden states through the imposition of constraints designed to achieve specific objectives, as illustrated through three examples: (1) the generation of adversarial-attack examples, (2) the usage of a neural network as a black-box optimization method, and (3) the application of inference on continuous-action reinforcement learning. These applications showcase how tasks that were previously reserved to gradient-based optimization approaches can now be approached with analytically tractable inference