Upper and lower bounds for the Lipschitz constant of random neural networks
This work addresses the problem of adversarial robustness for neural networks, providing theoretical bounds that are incremental to existing initialization and Lipschitz constant analyses.
The paper tackles the problem of quantifying worst-case robustness of neural networks against adversarial perturbations by studying upper and lower bounds for the Lipschitz constant of random ReLU networks with generalized He initialization. For deep networks of fixed depth and large width, the upper bound exceeds the lower bound by a logarithmic factor in width, while for shallow networks, the Lipschitz constant is characterized up to an absolute constant independent of parameters.
Empirical studies have widely demonstrated that neural networks are highly sensitive to small, adversarial perturbations of the input. The worst-case robustness against these so-called adversarial examples can be quantified by the Lipschitz constant of the neural network. In this paper, we study upper and lower bounds for the Lipschitz constant of random ReLU neural networks. Specifically, we assume that the weights and biases follow a generalization of the He initialization, where general symmetric distributions for the biases are permitted. For deep networks of fixed depth and sufficiently large width, our established upper bound is larger than the lower bound by a factor that is logarithmic in the width. In contrast, for shallow neural networks we characterize the Lipschitz constant up to an absolute numerical constant that is independent of all parameters.