Direct Parameterization of Lipschitz-Bounded Deep Networks
This work addresses the challenge of robust model training for security-critical applications like adversarial defense, offering a novel and efficient method.
The paper tackles the problem of ensuring Lipschitz bounds in deep neural networks to limit sensitivity to input perturbations, introducing a direct parameterization that matches the tightest-known SDP-based bounds and enables efficient training, with experiments showing improved robust accuracy in image classification tasks.
This paper introduces a new parameterization of deep neural networks (both fully-connected and convolutional) with guaranteed $\ell^2$ Lipschitz bounds, i.e. limited sensitivity to input perturbations. The Lipschitz guarantees are equivalent to the tightest-known bounds based on certification via a semidefinite program (SDP). We provide a ``direct'' parameterization, i.e., a smooth mapping from $\mathbb R^N$ onto the set of weights satisfying the SDP-based bound. Moreover, our parameterization is complete, i.e. a neural network satisfies the SDP bound if and only if it can be represented via our parameterization. This enables training using standard gradient methods, without any inner approximation or computationally intensive tasks (e.g. projections or barrier terms) for the SDP constraint. The new parameterization can equivalently be thought of as either a new layer type (the \textit{sandwich layer}), or a novel parameterization of standard feedforward networks with parameter sharing between neighbouring layers. A comprehensive set of experiments on image classification shows that sandwich layers outperform previous approaches on both empirical and certified robust accuracy. Code is available at \url{https://github.com/acfr/LBDN}.