EELBERT: Tiny Models through Dynamic Embeddings
This addresses the problem of deploying large models on resource-constrained devices, though it is incremental as it builds on existing compression techniques.
The paper tackles model compression for transformer-based models like BERT by replacing the input embedding layer with dynamic on-the-fly computations, resulting in minimal accuracy loss; for example, their smallest model achieves a GLUE score within 4% of BERT-tiny while being 15x smaller at 1.2 MB.
We introduce EELBERT, an approach for compression of transformer-based models (e.g., BERT), with minimal impact on the accuracy of downstream tasks. This is achieved by replacing the input embedding layer of the model with dynamic, i.e. on-the-fly, embedding computations. Since the input embedding layer accounts for a significant fraction of the model size, especially for the smaller BERT variants, replacing this layer with an embedding computation function helps us reduce the model size significantly. Empirical evaluation on the GLUE benchmark shows that our BERT variants (EELBERT) suffer minimal regression compared to the traditional BERT models. Through this approach, we are able to develop our smallest model UNO-EELBERT, which achieves a GLUE score within 4% of fully trained BERT-tiny, while being 15x smaller (1.2 MB) in size.