RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications
This work addresses the need for reliable robustness verification and training stability in neural networks, particularly for adversarial defense and GANs, though it is incremental as it builds on prior bounding techniques.
The paper tackles the problem of efficiently bounding the Jacobian matrix of neural networks to assess properties like Lipschitz constants and robustness, resulting in better-quality Lipschitz constants and improved robustness verification compared to previous methods, with polynomial time complexity and reasonable computation for large networks.
The Jacobian matrix (or the gradient for single-output networks) is directly related to many important properties of neural networks, such as the function landscape, stationary points, (local) Lipschitz constants and robustness to adversarial attacks. In this paper, we propose a recursive algorithm, RecurJac, to compute both upper and lower bounds for each element in the Jacobian matrix of a neural network with respect to network's input, and the network can contain a wide range of activation functions. As a byproduct, we can efficiently obtain a (local) Lipschitz constant, which plays a crucial role in neural network robustness verification, as well as the training stability of GANs. Experiments show that (local) Lipschitz constants produced by our method is of better quality than previous approaches, thus providing better robustness verification results. Our algorithm has polynomial time complexity, and its computation time is reasonable even for relatively large networks. Additionally, we use our bounds of Jacobian matrix to characterize the landscape of the neural network, for example, to determine whether there exist stationary points in a local neighborhood. Source code available at \url{http://github.com/huanzhang12/RecurJac-Jacobian-bounds}.