Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections
This work addresses uncertainty modeling for deep learning applications, offering a novel approach that enhances reliability in tasks like out-of-distribution detection, though it appears incremental as it builds on existing Bayesian neural network frameworks.
The paper tackles the problem of modeling uncertainty in deep neural networks by proposing a new architecture that shares neural connectivity patterns between generative and discriminative networks, leading to significant improvements in calibration and out-of-distribution detection compared to state-of-the-art methods.
Modeling uncertainty in deep neural networks, despite recent important advances, is still an open problem. Bayesian neural networks are a powerful solution, where the prior over network weights is a design choice, often a normal distribution or other distribution encouraging sparsity. However, this prior is agnostic to the generative process of the input data, which might lead to unwarranted generalization for out-of-distribution tested data. We suggest the presence of a confounder for the relation between the input data and the discriminative function given the target label. We propose an approach for modeling this confounder by sharing neural connectivity patterns between the generative and discriminative networks. This approach leads to a new deep architecture, where networks are sampled from the posterior of local causal structures, and coupled into a compact hierarchy. We demonstrate that sampling networks from this hierarchy, proportionally to their posterior, is efficient and enables estimating various types of uncertainties. Empirical evaluations of our method demonstrate significant improvement compared to state-of-the-art calibration and out-of-distribution detection methods.