A Complete Characterisation of ReLU-Invariant Distributions
This work addresses a foundational issue in Bayesian networks and neural network analysis for uncertainty quantification and explainable AI, but it is incremental as it builds on prior theoretical studies of invariance.
The paper tackles the problem of characterizing probability distributions invariant under ReLU neural network layers, proving that no invariant parametrised family exists without severe restrictions like impractical width, finite support, or non-Lipschitz parametrisation, and constructs examples for each case.
We give a complete characterisation of families of probability distributions that are invariant under the action of ReLU neural network layers. The need for such families arises during the training of Bayesian networks or the analysis of trained neural networks, e.g., in the context of uncertainty quantification (UQ) or explainable artificial intelligence (XAI). We prove that no invariant parametrised family of distributions can exist unless at least one of the following three restrictions holds: First, the network layers have a width of one, which is unreasonable for practical neural networks. Second, the probability measures in the family have finite support, which basically amounts to sampling distributions. Third, the parametrisation of the family is not locally Lipschitz continuous, which excludes all computationally feasible families. Finally, we show that these restrictions are individually necessary. For each of the three cases we can construct an invariant family exploiting exactly one of the restrictions but not the other two.