Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
This addresses the need for adaptable model compression in text generation tasks, offering a more flexible alternative to static compression methods, though it is incremental in building on existing knowledge distillation techniques.
The paper tackles the problem of compressing large pre-trained Transformer models for efficient inference by introducing Modular Transformers, a framework that trains modularized layers to replace multiple consecutive layers, enabling flexible compression ratios from 1.1x to 6x with minimal to moderate performance drops.
Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. Common neural network compression techniques such as knowledge distillation or quantization are limited to static compression where the compression ratio is fixed. In this paper, we introduce Modular Transformers, a modularized encoder-decoder framework for flexible sequence-to-sequence model compression. Modular Transformers train modularized layers that have the same function of two or more consecutive layers in the original model via module replacing and knowledge distillation. After training, the modularized layers can be flexibly assembled into sequence-to-sequence models that meet different performance-efficiency trade-offs. Experimental results show that after a single training phase, by simply varying the assembling strategy, Modular Transformers can achieve flexible compression ratios from 1.1x to 6x with little to moderate relative performance drop.