MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
This addresses the efficiency-accuracy tradeoff in deploying deep learning models on resource-constrained devices, representing an incremental improvement over existing quantization methods.
The paper tackles the accuracy degradation problem in quantized deep learning models by proposing MoQE, a Mixture of Quantization Experts framework that dynamically routes inputs to specialized quantization experts, achieving performance comparable to state-of-the-art quantization models on benchmark datasets like ImageNet and WikiText without significant latency increases.
Quantization method plays a crucial role in improving model efficiency and reducing deployment costs, enabling the widespread application of deep learning models on resource-constrained devices. However, the quantization process inevitably introduces accuracy degradation. In this paper, we propose Mixture of Quantization Experts( abbr. MoQE), a quantization inference framework based on the Mixture-of-Experts (MoE) architecture, aiming to jointly improve the performance of quantization models. MoQE combines multiple quantization variants of one full-precision model as specialized "quantization experts" and dynamically routes input data to the most suitable expert based on its characteristics. MoQE alleviates the performance degradation commonly seen in single quantization models through specialization quantization expert models. We design lightweight, structure-aware router models tailored for both CV and NLP tasks. Experimental evaluations on ResNet, LLaMA, and Qwen model families across benchmark datasets including ImageNet, WikiText, C4, and OpenWebText demonstrate that MoQE achieves performance comparable to SOTA quantization model, without incurring significant increases in inference latency.