Translating Numerical Concepts for PDEs into Neural Architectures
This work offers a numerical perspective on neural network design, potentially benefiting researchers in machine learning and computational mathematics, though it appears incremental as it builds on existing numerical methods.
The paper tackled the problem of understanding neural network architectures by translating numerical algorithms for PDEs into neural networks, resulting in connections that guarantee Euclidean stability for specific ResNets and provide design criteria for stable networks.
We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent networks, and U-nets. These connections guarantee Euclidean stability of specific ResNets with a transposed convolution layer structure in each block. We present three numerical justifications for skip connections: as time discretisations in explicit schemes, as extrapolation mechanisms for accelerating those methods, and as recurrent connections in fixed point solvers for implicit schemes. Last but not least, we also motivate uncommon design choices such as nonmonotone activation functions. Our findings give a numerical perspective on the success of modern neural network architectures, and they provide design criteria for stable networks.