Learning Trajectories are Generalization Indicators
This provides a novel perspective on generalization analysis for deep learning practitioners, though it appears incremental in advancing theoretical understanding rather than solving a practical bottleneck.
The paper tackles the problem of understanding generalization in deep neural networks by analyzing learning trajectories during training, proposing a new generalization bound based on trajectory complexity and training set characteristics. Experimental results show the method effectively captures generalization error throughout training and tracks changes with learning rate and label noise adjustments.
This paper explores the connection between learning trajectories of Deep Neural Networks (DNNs) and their generalization capabilities when optimized using (stochastic) gradient descent algorithms. Instead of concentrating solely on the generalization error of the DNN post-training, we present a novel perspective for analyzing generalization error by investigating the contribution of each update step to the change in generalization error. This perspective allows for a more direct comprehension of how the learning trajectory influences generalization error. Building upon this analysis, we propose a new generalization bound that incorporates more extensive trajectory information. Our proposed generalization bound depends on the complexity of learning trajectory and the ratio between the bias and diversity of training set. Experimental findings reveal that our method effectively captures the generalization error throughout the training process. Furthermore, our approach can also track changes in generalization error when adjustments are made to learning rates and label noise levels. These results demonstrate that learning trajectory information is a valuable indicator of a model's generalization capabilities.