On the distance between two neural networks and the stability of learning

This work addresses the challenge of training deeper and more complex neural networks more efficiently, potentially benefiting machine learning practitioners by reducing hyperparameter tuning efforts.

The paper tackles the problem of relating parameter distance to gradient breakdown in nonlinear compositional functions, resulting in a new distance function called deep relative trust and a descent lemma for neural networks, which may simplify training by requiring little to no learning rate tuning.