Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
This work addresses efficiency challenges for deploying language models in settings with limited computational resources, representing an incremental improvement over existing parameter-sharing methods.
The paper tackles the problem of high computational costs during inference in parameter-shared pre-trained language models, which limits their practicality in resource-constrained environments, and introduces techniques based on neural ODEs and pre-training to enhance inference efficiency, achieving significant acceleration in experiments on autoregressive and autoencoding models.
Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments, enabling substantial reductions in model storage and memory costs without significant performance compromise. However, it is important to note that parameter sharing does not alleviate computational burdens associated with inference, thus impeding its practicality in situations characterized by limited stringent latency requirements or computational resources. Building upon neural ordinary differential equations (ODEs), we introduce a straightforward technique to enhance the inference efficiency of parameter-shared PLMs. Additionally, we propose a simple pre-training technique that leads to fully or partially shared models capable of achieving even greater inference acceleration. The experimental results demonstrate the effectiveness of our methods on both autoregressive and autoencoding PLMs, providing novel insights into more efficient utilization of parameter-shared models in resource-constrained settings.