With Greater Distance Comes Worse Performance: On the Perspective of Layer Utilization and Model Generalization
This work addresses the problem of understanding and improving generalization in deep neural networks for machine learning researchers, offering incremental insights into layer-specific contributions and potential regularization techniques.
The paper investigates how different layers in neural networks contribute to generalization, finding that early layers learn broadly useful representations while deeper layers primarily minimize training risk and correlate with overfitting. It demonstrates that the distance of final layer weights from initialization correlates with generalization error and that re-initializing these weights can serve as a post-training regularization method.
Generalization of deep neural networks remains one of the main open problems in machine learning. Previous theoretical works focused on deriving tight bounds of model complexity, while empirical works revealed that neural networks exhibit double descent with respect to both training sample counts and the neural network size. In this paper, we empirically examined how different layers of neural networks contribute differently to the model; we found that early layers generally learn representations relevant to performance on both training data and testing data. Contrarily, deeper layers only minimize training risks and fail to generalize well with testing or mislabeled data. We further illustrate the distance of trained weights to its initial value of final layers has high correlation to generalization errors and can serve as an indicator of an overfit of model. Moreover, we show evidence to support post-training regularization by re-initializing weights of final layers. Our findings provide an efficient method to estimate the generalization capability of neural networks, and the insight of those quantitative results may inspire derivation to better generalization bounds that take the internal structure of neural networks into consideration.