Machine Learning, Deepest Learning: Statistical Data Assimilation Problems
This work provides a theoretical bridge between machine learning and data assimilation, potentially offering new optimization methods, but it is incremental as it builds on existing variational frameworks without demonstrating practical gains.
The paper establishes a formal equivalence between machine learning layers and time in statistical data assimilation, showing that deeper networks correspond to higher temporal resolution, and introduces 'deepest learning' as a continuous-layer framework leading to differential equations with symplectic symmetry.
We formulate a strong equivalence between machine learning, artificial intelligence methods and the formulation of statistical data assimilation as used widely in physical and biological sciences. The correspondence is that layer number in the artificial network setting is the analog of time in the data assimilation setting. Within the discussion of this equivalence we show that adding more layers (making the network deeper) is analogous to adding temporal resolution in a data assimilation framework. How one can find a candidate for the global minimum of the cost functions in the machine learning context using a method from data assimilation is discussed. Calculations on simple models from each side of the equivalence are reported. Also discussed is a framework in which the time or layer label is taken to be continuous, providing a differential equation, the Euler-Lagrange equation, which shows that the problem being solved is a two point boundary value problem familiar in the discussion of variational methods. The use of continuous layers is denoted "deepest learning". These problems respect a symplectic symmetry in continuous time/layer phase space. Both Lagrangian versions and Hamiltonian versions of these problems are presented. Their well-studied implementation in a discrete time/layer, while respected the symplectic structure, is addressed. The Hamiltonian version provides a direct rationale for back propagation as a solution method for the canonical momentum.