Extracting Global Dynamics of Loss Landscape in Deep Learning Models
This provides an empirical framework for understanding and improving training stability in deep learning, though it appears incremental as it builds on existing theoretical landscape characterizations.
The authors tackled the problem of inconsistent predictions from deep learning models by developing DOODL3, a toolkit that formulates training as a dynamical system to extract global dynamics from the loss landscape, enabling interpretable analysis and guidance for neural network training.
Deep learning models evolve through training to learn the manifold in which the data exists to satisfy an objective. It is well known that evolution leads to different final states which produce inconsistent predictions of the same test data points. This calls for techniques to be able to empirically quantify the difference in the trajectories and highlight problematic regions. While much focus is placed on discovering what models learn, the question of how a model learns is less studied beyond theoretical landscape characterizations and local geometric approximations near optimal conditions. Here, we present a toolkit for the Dynamical Organization Of Deep Learning Loss Landscapes, or DOODL3. DOODL3 formulates the training of neural networks as a dynamical system, analyzes the learning process, and presents an interpretable global view of trajectories in the loss landscape. Our approach uses the coarseness of topology to capture the granularity of geometry to mitigate against states of instability or elongated training. Overall, our analysis presents an empirical framework to extract the global dynamics of a model and to use that information to guide the training of neural networks.