Lipschitz constant estimation for general neural network architectures using control tools
This work addresses the need for scalable and generalizable Lipschitz constant estimation in neural networks, which is crucial for robustness and stability analysis in machine learning, but it appears incremental by building on prior control theory methods.
The paper tackles the problem of estimating the Lipschitz constant for general neural network architectures by interpreting them as time-varying dynamical systems and using semidefinite programming with dynamic programming recursion. It demonstrates computational advantages and versatility by applying the method to networks trained on MNIST and CIFAR-10, though no specific numerical results are provided.
This paper is devoted to the estimation of the Lipschitz constant of general neural network architectures using semidefinite programming. For this purpose, we interpret neural networks as time-varying dynamical systems, where the $k$-th layer corresponds to the dynamics at time $k$. A key novelty with respect to prior work is that we use this interpretation to exploit the series interconnection structure of feedforward neural networks with a dynamic programming recursion. Nonlinearities, such as activation functions and nonlinear pooling layers, are handled with integral quadratic constraints. If the neural network contains signal processing layers (convolutional or state space model layers), we realize them as 1-D/2-D/N-D systems and exploit this structure as well. We distinguish ourselves from related work on Lipschitz constant estimation by more extensive structure exploitation (scalability) and a generalization to a large class of common neural network architectures. To show the versatility and computational advantages of our method, we apply it to different neural network architectures trained on MNIST and CIFAR-10.