Neural Nets via Forward State Transformation and Backward Loss Transformation
This provides a theoretical, incremental perspective on neural networks for researchers in machine learning and formal methods.
The paper reinterprets neural network training by viewing forward passes as state transformations and backward passes as loss transformations, aligning with program semantics, and demonstrates this with a simple training example.
This article studies (multilayer perceptron) neural networks with an emphasis on the transformations involved --- both forward and backward --- in order to develop a semantical/logical perspective that is in line with standard program semantics. The common two-pass neural network training algorithms make this viewpoint particularly fitting. In the forward direction, neural networks act as state transformers. In the reverse direction, however, neural networks change losses of outputs to losses of inputs, thereby acting like a (real-valued) predicate transformer. In this way, backpropagation is functorial by construction, as shown earlier in recent other work. We illustrate this perspective by training a simple instance of a neural network.