Analog Bayesian neural networks are insensitive to the shape of the weight distribution
This work addresses hardware designers by showing that analog BNN implementations can be insensitive to noise distribution shapes, potentially simplifying design and reducing energy consumption, though it is incremental as it builds on existing MFVI methods.
The paper tackles the challenge of implementing Bayesian neural networks (BNNs) with mean field variational inference (MFVI) in analog hardware, where controlling noise distribution shapes is difficult, and demonstrates that predictive distributions converge regardless of noise shape, enabling energy-efficient designs without shape constraints.
Recent work has demonstrated that Bayesian neural networks (BNN's) trained with mean field variational inference (MFVI) can be implemented in analog hardware, promising orders of magnitude energy savings compared to the standard digital implementations. However, while Gaussians are typically used as the variational distribution in MFVI, it is difficult to precisely control the shape of the noise distributions produced by sampling analog devices. This paper introduces a method for MFVI training using real device noise as the variational distribution. Furthermore, we demonstrate empirically that the predictive distributions from BNN's with the same weight means and variances converge to the same distribution, regardless of the shape of the variational distribution. This result suggests that analog device designers do not need to consider the shape of the device noise distribution when hardware-implementing BNNs performing MFVI.