Enhancing Neural Training via a Correlated Dynamics Model
This addresses the problem of high computational and communication costs in training large-scale neural networks, particularly in federated learning, with incremental improvements over existing methods.
The paper tackles the computational demands of training large neural networks by introducing Correlation Mode Decomposition (CMD), which clusters parameters into synchronized modes to efficiently model training dynamics, resulting in improved test set generalization and surpassing state-of-the-art methods in image classification.
As neural networks grow in scale, their training becomes both computationally demanding and rich in dynamics. Amidst the flourishing interest in these training dynamics, we present a novel observation: Parameters during training exhibit intrinsic correlations over time. Capitalizing on this, we introduce Correlation Mode Decomposition (CMD). This algorithm clusters the parameter space into groups, termed modes, that display synchronized behavior across epochs. This enables CMD to efficiently represent the training dynamics of complex networks, like ResNets and Transformers, using only a few modes. Moreover, test set generalization is enhanced. We introduce an efficient CMD variant, designed to run concurrently with training. Our experiments indicate that CMD surpasses the state-of-the-art method for compactly modeled dynamics on image classification. Our modeling can improve training efficiency and lower communication overhead, as shown by our preliminary experiments in the context of federated learning.