Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias
For deep learning practitioners, this work reveals that the practical inductive bias of a trained network is not just the architectural prior but is filtered by the forgetting dynamics of the training pipeline, with regularization methods that improve generalization also erasing initialization memory.
The paper investigates how much of a neural network's initial random bias survives training, introducing 'initialization memory' as a metric. Experiments on CIFAR-10 with ResNets show that low-learning-rate SGD can achieve high training accuracy while retaining strong dependence on initialization (test accuracy varies by 26.5 percentage points), whereas Adam and larger learning rates with L2 regularization erase this memory.
Randomly initialized neural networks induce a prior over functions, but the predictor used in practice is produced only after training. We ask how much of this initial bias survives the training pipeline. To make the question measurable, we introduce initialization memory: the dependence of the validation-selected predictor on the scale of the random initialization. We perform controlled CIFAR-10 experiments on ResNets where initialization memory already sharply separates training regimes. Low-learning-rate SGD can interpolate while still remembering its initialization: on ResNet-9 with batch size $b=128$, test accuracy varies by $26.5$ percentage points across initialization scales despite $\ge99.5\%$ training accuracy. This is not undertraining: extending the same low-learning-rate regime to $5{,}000$ epochs leaves the spread essentially unchanged. In contrast, Adam-family methods largely erase the dependence. SGD can also be made to forget when larger learning rates are paired with explicit $L_2$ norm control. We interpret these findings in terms of the time scale of forgetting: gradient-flow-like dynamics can preserve initialization memory, whereas stochastic finite-step effects, explicit norm decay, and adaptive preconditioning erase it on scales governed by the size of explicit or implicit regularization. The practical inductive bias of a trained network is therefore not the architectural prior alone, but the architectural prior after being filtered by the forgetting dynamics of the training pipeline; and the same regularizers that improve generalization are precisely those that erase memory of initialization.