Designing Stable Neural Networks using Convex Analysis and ODEs
This work addresses stability issues in neural networks for tasks like robust classification and image processing, offering a novel architectural approach with potential broad impact, though it builds incrementally on classical ODE and convex analysis methods.
The authors tackled the problem of designing stable neural networks by proposing a ResNet-style architecture that ensures non-expansive (1-Lipschitz) operators through spectral norm constraints on weights, contrasting with ordinary ResNets that can have exponentially growing Lipschitz constants. They applied this architecture to adversarially robust image classification, image denoising, and deblurring, showing it can be trained performantly with these constraints.
Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further analysis of the proposed architecture shows that the spectral norms of the weights can be further constrained to ensure that the network is an averaged operator, making it a natural candidate for a learned denoiser in Plug-and-Play algorithms. Using a novel adaptive way of enforcing the spectral norm constraints, we show that, even with these constraints, it is possible to train performant networks. The proposed architecture is applied to the problem of adversarially robust image classification, to image denoising, and finally to the inverse problem of deblurring.