Early alignment in two-layer networks training is a two-edged sword
This work addresses the trade-off between sparsity and convergence in neural network training, which is incremental but relevant for researchers optimizing deep learning algorithms.
The paper analyzes the early alignment phase in two-layer ReLU networks with small initialization, showing that it induces a sparse representation linked to gradient flow's implicit bias. However, this sparsity can hinder training, as demonstrated by an example where overparameterized networks fail to reach global minima and converge to spurious stationary points.
Training neural networks with first order optimisation methods is at the core of the empirical success of deep learning. The scale of initialisation is a crucial factor, as small initialisations are generally associated to a feature learning regime, for which gradient descent is implicitly biased towards simple solutions. This work provides a general and quantitative description of the early alignment phase, originally introduced by Maennel et al. (2018). For small initialisation and one hidden ReLU layer networks, the early stage of the training dynamics leads to an alignment of the neurons towards key directions. This alignment induces a sparse representation of the network, which is directly related to the implicit bias of gradient flow at convergence. This sparsity inducing alignment however comes at the expense of difficulties in minimising the training objective: we also provide a simple data example for which overparameterised networks fail to converge towards global minima and only converge to a spurious stationary point instead.