Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
This addresses the deployment difficulty of slow Transformer models for natural language processing applications, representing an incremental improvement in optimization.
The paper tackled the problem of slow inference in Transformer models by developing an efficient one-shot neural architecture search algorithm that decomposes the architecture into components, achieving 10-30% speedup for pre-trained BERT and 70% speedup on a distilled BERT model with acceptable performance drops.
Transformer-based models have achieved stateof-the-art results in many tasks in natural language processing. However, such models are usually slow at inference time, making deployment difficult. In this paper, we develop an efficient algorithm to search for fast models while maintaining model quality. We describe a novel approach to decompose the Transformer architecture into smaller components, and propose a sampling-based one-shot architecture search method to find an optimal model for inference. The model search process is more efficient than alternatives, adding only a small overhead to training time. By applying our methods to BERT-base architectures, we achieve 10% to 30% speedup for pre-trained BERT and 70% speedup on top of a previous state-of-the-art distilled BERT model on Cloud TPU-v2 with a generally acceptable drop in performance.