On the Epistemic Uncertainty of Overparametrized Neural Networks
For researchers in Bayesian deep learning, it clarifies that epistemic uncertainty in overparametrized networks is not purely reducible, challenging common assumptions.
The paper analyzes epistemic uncertainty in overparametrized neural networks, showing that non-identifiability leads to residual parameter uncertainty even when the underlying function is fully identified. Theoretical analysis on one-hidden-layer ReLU networks is validated empirically.
Epistemic uncertainty is often viewed as a reducible uncertainty that vanishes with increasing data. This perspective implicitly assumes parameter identifiability and equates epistemic uncertainty with predictive variability. In overparametrized neural networks, however, model parameters are typically non-identifiable due to symmetries and redundant representations. As a consequence, substantial parameter uncertainty can persist even when the underlying function is fully identified. In this work, we analyze epistemic uncertainty through the lens of non-identifiability and characterize both discrete and continuous sources of residual uncertainty. Focusing on one-hidden-layer ReLU networks, we thoroughly analyze the resulting posterior structure and validate our theoretical insights through empirical studies.