On the Variance of Neural Network Training with respect to Test Sets and Distributions
This work addresses reproducibility and hyperparameter comparison issues in neural network training, offering insights into variance sources, but it is incremental as it builds on prior calibration findings.
The paper tackles the problem of high variance in neural network test-set performance across training runs, showing that while variance on specific test-sets is substantial, variance on the underlying test-distributions is minimal, and errors on test examples are approximately independent. It provides a formula predicting variance for binary classification and applies this analysis to factors like data augmentation and distribution shift.
Typical neural network trainings have substantial variance in test-set performance between repeated runs, impeding hyperparameter comparison and training reproducibility. In this work we present the following results towards understanding this variation. (1) Despite having significant variance on their test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have little variance in performance on the underlying test-distributions from which their test-sets are sampled. (2) We show that these trainings make approximately independent errors on their test-sets. That is, the event that a trained network makes an error on one particular example does not affect its chances of making errors on other examples, relative to their average rates over repeated runs of training with the same hyperparameters. (3) We prove that the variance of neural network trainings on their test-sets is a downstream consequence of the class-calibration property discovered by Jiang et al. (2021). Our analysis yields a simple formula which accurately predicts variance for the binary classification case. (4) We conduct preliminary studies of data augmentation, learning rate, finetuning instability and distribution-shift through the lens of variance between runs.