Deep neural networks from the perspective of ergodic theory
This work provides a theoretical foundation for understanding deep learning design rules, which is incremental as it builds on existing dynamical systems perspectives.
The paper tackled the problem of deep neural network design being more art than science by applying ergodic theory to view networks as dynamical systems, resulting in the attribution of some design heuristics to theoretical principles.
The design of deep neural networks remains somewhat of an art rather than precise science. By tentatively adopting ergodic theory considerations on top of viewing the network as the time evolution of a dynamical system, with each layer corresponding to a temporal instance, we show that some rules of thumb, which might otherwise appear mysterious, can be attributed heuristics.