Lipschitz Bounded Equilibrium Networks
This work addresses robustness in neural networks for applications like image classification, but it is incremental as it builds on existing equilibrium network frameworks.
The paper tackled the problem of ensuring Lipschitz bounds in equilibrium neural networks for robustness and generalization, achieving accurate bounds and improved adversarial robustness in image classification experiments.
This paper introduces new parameterizations of equilibrium neural networks, i.e. networks defined by implicit equations. This model class includes standard multilayer and residual networks as special cases. The new parameterization admits a Lipschitz bound during training via unconstrained optimization: no projections or barrier functions are required. Lipschitz bounds are a common proxy for robustness and appear in many generalization bounds. Furthermore, compared to previous works we show well-posedness (existence of solutions) under less restrictive conditions on the network weights and more natural assumptions on the activation functions: that they are monotone and slope restricted. These results are proved by establishing novel connections with convex optimization, operator splitting on non-Euclidean spaces, and contracting neural ODEs. In image classification experiments we show that the Lipschitz bounds are very accurate and improve robustness to adversarial attacks.