Infinitely deep neural networks as diffusion processes
This addresses stability and function representation issues in deep learning for researchers and practitioners, though it appears incremental as it builds on existing work on infinite-depth networks.
The paper tackles the problem of undesirable properties in infinitely deep neural networks, such as vanishing input dependency, by linking them to diffusion processes via parameter distributions that shrink with depth, showing these limiting processes avoid those issues.
When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parameter distributions that shrink as the number of layers increases in order to recover well-behaved stochastic processes in the limit of infinite depth. This leads to set forth a link between infinitely deep residual networks and solutions to stochastic differential equations, i.e. diffusion processes. We show that these limiting processes do not suffer from the aforementioned issues and investigate their properties.