Translating Diffusion, Wavelets, and Regularisation into Residual Networks
This work addresses stability issues in CNNs for researchers and practitioners in machine learning and computer vision, offering a bridge between classical and modern approaches, though it is incremental in applying existing methods to new contexts.
The paper tackled the problem of understanding and improving the stability of convolutional neural networks (CNNs) by translating classical signal denoising methods (nonlinear diffusion, wavelets, and regularization) into residual network architectures, resulting in new nonmonotone activation functions and intrinsically stable CNN designs.
Convolutional neural networks (CNNs) often perform well, but their stability is poorly understood. To address this problem, we consider the simple prototypical problem of signal denoising, where classical approaches such as nonlinear diffusion, wavelet-based methods and regularisation offer provable stability guarantees. To transfer such guarantees to CNNs, we interpret numerical approximations of these classical methods as a specific residual network (ResNet) architecture. This leads to a dictionary which allows to translate diffusivities, shrinkage functions, and regularisers into activation functions, and enables a direct communication between the four research communities. On the CNN side, it does not only inspire new families of nonmonotone activation functions, but also introduces intrinsically stable architectures for an arbitrary number of layers.