It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Models
This work provides a new theoretical insight into generalization error in deep learning, potentially influencing model design and evaluation for researchers and practitioners, though it is incremental in revisiting established concepts.
The paper challenges the classical bias-variance trade-off in machine learning by showing that for ensembles of deep learning classification models, bias and variance are aligned at a sample level, with squared bias approximately equal to variance for correctly classified points, supported by empirical evidence across various models and datasets.
Classical wisdom in machine learning holds that the generalization error can be decomposed into bias and variance, and these two terms exhibit a \emph{trade-off}. However, in this paper, we show that for an ensemble of deep learning based classification models, bias and variance are \emph{aligned} at a sample level, where squared bias is approximately \emph{equal} to variance for correctly classified sample points. We present empirical evidence confirming this phenomenon in a variety of deep learning models and datasets. Moreover, we study this phenomenon from two theoretical perspectives: calibration and neural collapse. We first show theoretically that under the assumption that the models are well calibrated, we can observe the bias-variance alignment. Second, starting from the picture provided by the neural collapse theory, we show an approximate correlation between bias and variance.