BabyBear: Cheap inference triage for expensive language models
This addresses the problem of computational and environmental expense for users of large language models, though it is incremental as it builds on existing cascading concepts from computer vision.
The paper tackles the high computational cost of transformer language models by introducing BabyBear, a framework for cascading models with inference triage to exit early when cheap models achieve high-confidence predictions, reducing compute cost by over 50% for classification and 33% for named entity recognition while maintaining accuracy.
Transformer language models provide superior accuracy over previous models but they are computationally and environmentally expensive. Borrowing the concept of model cascading from computer vision, we introduce BabyBear, a framework for cascading models for natural language processing (NLP) tasks to minimize cost. The core strategy is inference triage, exiting early when the least expensive model in the cascade achieves a sufficiently high-confidence prediction. We test BabyBear on several open source data sets related to document classification and entity recognition. We find that for common NLP tasks a high proportion of the inference load can be accomplished with cheap, fast models that have learned by observing a deep learning model. This allows us to reduce the compute cost of large-scale classification jobs by more than 50% while retaining overall accuracy. For named entity recognition, we save 33% of the deep learning compute while maintaining an F1 score higher than 95% on the CoNLL benchmark.