Dependence between Bayesian neural network units
This work provides insights into the practical behavior of Bayesian neural networks, which is incremental but important for researchers and practitioners in machine learning.
The paper investigates the dependence properties of hidden units in finite-width Bayesian neural networks, addressing the gap between theoretical infinite-width limits and practical implementations, and empirically evaluates how depth and width affect these dependencies.
The connection between Bayesian neural networks and Gaussian processes gained a lot of attention in the last few years, with the flagship result that hidden units converge to a Gaussian process limit when the layers width tends to infinity. Underpinning this result is the fact that hidden units become independent in the infinite-width limit. Our aim is to shed some light on hidden units dependence properties in practical finite-width Bayesian neural networks. In addition to theoretical results, we assess empirically the depth and width impacts on hidden units dependence properties.