Uncertainty propagation in feed-forward neural network models
This work addresses uncertainty quantification in neural networks for domains requiring robust predictions, though it is incremental as it builds on existing methods for specific architectures.
The paper tackled the problem of propagating input uncertainty through feed-forward neural networks with leaky ReLU activations, deriving analytical expressions for the output probability density function and statistical moments, and demonstrating excellent agreement with Monte Carlo simulations in validation tests.
We develop new uncertainty propagation methods for feed-forward neural network architectures with leaky ReLU activation functions subject to random perturbations in the input vectors. In particular, we derive analytical expressions for the probability density function (PDF) of the neural network output and its statistical moments as a function of the input uncertainty and the parameters of the network, i.e., weights and biases. A key finding is that an appropriate linearization of the leaky ReLU activation function yields accurate statistical results even for large perturbations in the input vectors. This can be attributed to the way information propagates through the network. We also propose new analytically tractable Gaussian copula surrogate models to approximate the full joint PDF of the neural network output. To validate our theoretical results, we conduct Monte Carlo simulations and a thorough error analysis on a multi-layer neural network representing a nonlinear integro-differential operator between two polynomial function spaces. Our findings demonstrate excellent agreement between the theoretical predictions and Monte Carlo simulations.