Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models
This work addresses the challenge of efficient deployment of multilingual models, though it appears incremental as it builds on existing quantization techniques.
The paper tackles the problem of compressing Transformer-based language models via quantization, presenting a new method called self-distilled quantization (SDQ) that reduces models from 32-bit to 8-bit weights while maintaining high performance on the XGLUE benchmark.
We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models. We present a new method called self-distilled quantization (SDQ) that minimizes accumulative quantization errors and outperforms baselines. We apply SDQ to multilingual models XLM-R-Base and InfoXLM-Base and demonstrate that both models can be reduced from 32-bit floating point weights to 8-bit integer weights while maintaining a high level of performance on the XGLUE benchmark. Our results also highlight the challenges of quantizing multilingual models, which must generalize to languages they were not fine-tuned on.