A Flow Model of Neural Networks
This work provides a theoretical framework for interpreting neural network behaviors, which is incremental in applying differential equation tools to deep learning.
The authors tackled the problem of understanding deep neural networks by proposing a continuous flow model that connects ResNets and plain nets to transport equations, enabling explicit construction of ResNets as refinements of plain nets and explaining phenomena like the necessity of 2-layer blocks and depth benefits.
Based on a natural connection between ResNet and transport equation or its characteristic equation, we propose a continuous flow model for both ResNet and plain net. Through this continuous model, a ResNet can be explicitly constructed as a refinement of a plain net. The flow model provides an alternative perspective to understand phenomena in deep neural networks, such as why it is necessary and sufficient to use 2-layer blocks in ResNets, why deeper is better, and why ResNets are even deeper, and so on. It also opens a gate to bring in more tools from the huge area of differential equations.