Predicting on the Edge: Identifying Where a Larger Model Does Better
This work addresses the efficiency and cost challenges in deploying large models for practitioners by enabling selective use of larger models only when beneficial.
The paper tackles the problem of identifying which examples benefit from larger models, finding that large models improve most on examples where small models are uncertain, and demonstrates that a switcher model deferring uncertain examples to a larger model achieves significant performance and resource usage improvements, with concrete gains shown in numerical studies.
Much effort has been devoted to making large and more accurate models, but relatively little has been put into understanding which examples are benefiting from the added complexity. In this paper, we demonstrate and analyze the surprisingly tight link between a model's predictive uncertainty on individual examples and the likelihood that larger models will improve prediction on them. Through extensive numerical studies on the T5 encoder-decoder architecture, we show that large models have the largest improvement on examples where the small model is most uncertain. On more certain examples, even those where the small model is not particularly accurate, large models are often unable to improve at all, and can even perform worse than the smaller model. Based on these findings, we show that a switcher model which defers examples to a larger model when a small model is uncertain can achieve striking improvements in performance and resource usage. We also explore committee-based uncertainty metrics that can be more effective but less practical.