Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Services
This addresses the problem of slow and costly GPU inference for BERT-like models in online services like text mining and web searching, offering a domain-specific incremental improvement.
The paper tackles the inefficient online inference of large BERT-like models on GPUs by proposing a system that uses stacking distillation and boosting ensemble to create parallel shallow student models, achieving up to 4.1x lower latency and 22.27x higher throughput while maintaining accuracy.
Due to high accuracy, BERT-like models have been widely adopted by text mining and web searching. However, large BERT-like models suffer from inefficient online inference, facing the following two problems on GPUs: (1) their high accuracy relies on the large model depth, which linearly increases the sequential computation on GPUs; (2) stochastic and dynamic online workloads cause extra costs from batching and paddings. Therefore, we present \sys for the real-world setting of GPU inference on online workloads. At its core, \sys adopts stacking distillation and boosting ensemble, distilling the original deep model into a group of shallow but virtually stacked student models running in parallel. This enables \sys to achieve a lower model depth (e.g., two layers) than the others and the lowest inference latency while maintaining accuracy. In addition, adaptive student pruning realizes dynamic student numbers according to changing online workloads. Especially for occasional workload bursts, it can temporarily decrease the student number with minimal accuracy loss to improve system throughput. We conduct comprehensive experiments to verify the effectiveness, whose results show that \sys outperforms the baselines by $4.1\times\sim 1.6\times$ in latency while maintaining accuracy and achieves up to $22.27\times$ higher throughput for workload bursts.