No One Representation to Rule Them All: Overlapping Features of Training Methods
This work addresses the problem of limited ensemble benefits in machine learning by demonstrating that diverse training methods can improve model performance, though it is incremental as it builds on existing training techniques.
The study challenges the assumption that high-accuracy models learn similar functions by showing that models with different training methods produce uncorrelated errors and specialize in data subdomains, leading to a 7% boost in ensemble accuracy (e.g., from 76.5% to 83.4% on ImageNet).
Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks. This motivates us to revisit the assumption that models necessarily learn similar functions. We conduct a large-scale empirical study of models across hyper-parameters, architectures, frameworks, and datasets. We find that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors. We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 models (each with ImageNet accuracy ~76.5%), we can create ensembles with 83.4% (+7% boost). Surprisingly, we find that even significantly low-accuracy models can be used to improve high-accuracy models. Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.