The Vanishing Decision Boundary Complexity and the Strong First Component
This addresses the challenge of understanding generalization in deep learning by showing that decision boundary complexity is not persistent, offering a new approach using predecessor models, though it is incremental in scope.
The study reveals that complex decision boundary structures vanish early in deep model training, unlike in traditional machine learning, but predecessor models' boundaries can predict final generalization, with findings on the first principal component's strength, optimizer singularity, and ResNet skip connections.
We show that unlike machine learning classifiers, there are no complex boundary structures in the decision boundaries for well-trained deep models. However, we found that the complicated structures do appear in training but they vanish shortly after shaping. This is a pessimistic news if one seeks to capture different levels of complexity in the decision boundary for understanding generalization, which works well in machine learning. Nonetheless, we found that the decision boundaries of predecessor models on the training data are reflective of the final model's generalization. We show how to use the predecessor decision boundaries for studying the generalization of deep models. We have three major findings. One is on the strength of the first principle component of deep models, another about the singularity of optimizers, and the other on the effects of the skip connections in ResNets. Code is at https://github.com/hengshu1/decision_boundary_github.