Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones
This work addresses the challenge of deploying large models efficiently for inference, offering a method to scale up performance without increasing costs, which is incremental but practical for resource-constrained applications.
The paper tackles the problem of controlling the performance-compute trade-off in early exiting for large models, showing that PCEE enables better accuracy control and allows larger models to achieve higher performance at similar computational costs as smaller ones.
Early Exiting (EE) is a promising technique for speeding up inference by adaptively allocating compute resources to data points based on their difficulty. The approach enables predictions to exit at earlier layers for simpler samples while reserving more computation for challenging ones. In this study, we first present a novel perspective on the EE approach, showing that larger models deployed with EE can achieve higher performance than smaller models while maintaining similar computational costs. As existing EE approaches rely on confidence estimation at each exit point, we further study the impact of overconfidence on the controllability of the compute-performance trade-off. We introduce Performance Control Early Exiting (PCEE), a method that enables accuracy thresholding by basing decisions not on a data point's confidence but on the average accuracy of samples with similar confidence levels from a held-out validation set. In our experiments, we show that PCEE offers a simple yet computationally efficient approach that provides better control over performance than standard confidence-based approaches, and allows us to scale up model sizes to yield performance gain while reducing the computational cost.