Bayesian Neural Networks with Soft Evidence
This work addresses the challenge of handling uncertain evidence in Bayesian neural networks, which is an incremental advancement for machine learning practitioners dealing with noisy or ambiguous data.
The paper tackles the problem of learning Bayesian neural networks with soft evidence, where there is uncertainty about event occurrences, by proposing two algorithms based on Jeffrey's rule for approximating Jeffrey conditionalization. The result shows that these methods are competitive or better in accuracy and improve calibration metrics by up to 20% in some cases, even with mislabeled data.
Bayes's rule deals with hard evidence, that is, we can calculate the probability of event $A$ occuring given that event $B$ has occurred. Soft evidence, on the other hand, involves a degree of uncertainty about whether event $B$ has actually occurred or not. Jeffrey's rule of conditioning provides a way to update beliefs in the case of soft evidence. We provide a framework to learn a probability distribution on the weights of a neural network trained using soft evidence by way of two simple algorithms for approximating Jeffrey conditionalization. We propose an experimental protocol for benchmarking these algorithms on empirical datasets and find that Jeffrey based methods are competitive or better in terms of accuracy yet show improvements in calibration metrics upwards of 20% in some cases, even when the data contains mislabeled points.