Asymmetric Duos: Sidekicks Improve Uncertainty
This addresses the problem of high computational costs for uncertainty quantification in large models, offering a cost-effective solution for practitioners, though it is incremental as it builds on existing ensembling and model combination techniques.
The paper tackles the computational inefficiency of ensembling for uncertainty quantification in large-scale models by introducing Asymmetric Duos, which pair a large model with a smaller sidekick model, achieving significant improvements in accuracy, uncertainty quantification, and selective classification metrics with only about 10-20% more computation across five image classification benchmarks.
The go-to strategy to apply deep networks in settings where uncertainty informs decisions--ensembling multiple training runs with random initializations--is ill-suited for the extremely large-scale models and practical fine-tuning workflows of today. We introduce a new cost-effective strategy for improving the uncertainty quantification and downstream decisions of a large model (e.g. a fine-tuned ViT-B): coupling it with a less accurate but much smaller "sidekick" (e.g. a fine-tuned ResNet-34) with a fraction of the computational cost. We propose aggregating the predictions of this \emph{Asymmetric Duo} by simple learned weighted averaging. Surprisingly, despite their inherent asymmetry, the sidekick model almost never harms the performance of the larger model. In fact, across five image classification benchmarks and a variety of model architectures and training schemes (including soups), Asymmetric Duos significantly improve accuracy, uncertainty quantification, and selective classification metrics with only ${\sim}10-20\%$ more computation.