Exploring Low-Cost Transformer Model Compression for Large-Scale Commercial Reply Suggestions
This work addresses training efficiency for large-scale commercial applications, but it is incremental as it applies existing compression techniques to a specific domain.
The paper tackled the problem of unsustainable training times for fine-tuning pre-trained language models in commercial reply suggestion systems by exploring low-cost model compression techniques like Layer Dropping and Layer Freezing, resulting in a 42% reduction in training time without affecting model relevance or user engagement.
Fine-tuning pre-trained language models improves the quality of commercial reply suggestion systems, but at the cost of unsustainable training times. Popular training time reduction approaches are resource intensive, thus we explore low-cost model compression techniques like Layer Dropping and Layer Freezing. We demonstrate the efficacy of these techniques in large-data scenarios, enabling the training time reduction for a commercial email reply suggestion system by 42%, without affecting the model relevance or user engagement. We further study the robustness of these techniques to pre-trained model and dataset size ablation, and share several insights and recommendations for commercial applications.