Experiments with Rich Regime Training for Deep Learning
This work addresses efficiency and generalization in deep learning training, offering a method to speed up training without sacrificing performance, though it is incremental as it builds on existing sparse and layer-wise update techniques.
The paper investigates rich regime training in deep learning, finding that a small number of active parameters, often in bottom layers, are crucial for generalization, and proposes a probabilistic layer-wise sparse SGD method that matches vanilla SGD performance while reducing back-propagation time by 2-5 times.
In spite of advances in understanding lazy training, recent work attributes the practical success of deep learning to the rich regime with complex inductive bias. In this paper, we study rich regime training empirically with benchmark datasets, and find that while most parameters are lazy, there is always a small number of active parameters which change quite a bit during training. We show that re-initializing (resetting to their initial random values) the active parameters leads to worse generalization. Further, we show that most of the active parameters are in the bottom layers, close to the input, especially as the networks become wider. Based on such observations, we study static Layer-Wise Sparse (LWS) SGD, which only updates some subsets of layers. We find that only updating the top and bottom layers have good generalization and, as expected, only updating the top layers yields a fast algorithm. Inspired by this, we investigate probabilistic LWS-SGD, which mostly updates the top layers and occasionally updates the full network. We show that probabilistic LWS-SGD matches the generalization performance of vanilla SGD and the back-propagation time can be 2-5 times more efficient.