Machines of finite depth: towards a formalization of neural networks
This work provides a foundational formalization for neural network architectures, which could benefit researchers in machine learning theory and algorithm design.
The authors tackled the lack of a formal mathematical foundation for neural networks by introducing 'machines of finite depth' as a unifying framework, proving that these machines are modular, efficiently computable, and differentiable, with backward passes computable without overhead.
We provide a unifying framework where artificial neural networks and their architectures can be formally described as particular cases of a general mathematical construction--machines of finite depth. Unlike neural networks, machines have a precise definition, from which several properties follow naturally. Machines of finite depth are modular (they can be combined), efficiently computable and differentiable. The backward pass of a machine is again a machine and can be computed without overhead using the same procedure as the forward pass. We prove this statement theoretically and practically, via a unified implementation that generalizes several classical architectures--dense, convolutional, and recurrent neural networks with a rich shortcut structure--and their respective backpropagation rules.