Layer rotation: a surprisingly powerful indicator of generalization in deep networks?
This provides a practical tool for hyperparameter tuning and understanding generalization in deep learning, though it is incremental as it builds on existing empirical analysis of training dynamics.
The paper investigates layer rotation, the cosine distance between a layer's weights and its initialization during training, as an indicator of generalization in deep networks, finding that larger distances correlate with better performance, with an optimal distance of 1 improving test accuracy by up to 30%.
Our work presents extensive empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. In particular, larger cosine distances between final and initial weights of each layer consistently translate into better generalization performance of the final model. Interestingly, this relation admits a network independent optimum: training procedures during which all layers' weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 30% test accuracy. Moreover, we show that layer rotations are easily monitored and controlled (helpful for hyperparameter tuning) and potentially provide a unified framework to explain the impact of learning rate tuning, weight decay, learning rate warmups and adaptive gradient methods on generalization and training speed. In an attempt to explain the surprising properties of layer rotation, we show on a 1-layer MLP trained on MNIST that layer rotation correlates with the degree to which features of intermediate layers have been trained.