Mixture of Experts with Mixture of Precisions for Tuning Quality of Service
This work addresses the problem of flexible and efficient model deployment for users in multi-tenant, resource-varying settings, representing an incremental improvement in optimization techniques.
The paper tackles the challenge of efficiently deploying large Mixture-of-Experts models in resource-constrained environments by introducing an adaptive serving approach that uses partial quantization of experts to dynamically balance throughput and model quality, achieving throughput adjustments from 0.63 to 13.00 tokens per second with marginal perplexity increases of up to 0.58 across benchmarks.
The increasing demand for deploying large Mixture-of-Experts (MoE) models in resource-constrained environments necessitates efficient approaches to address their high memory and computational requirements challenges. Moreover, given that tasks come in different user-defined constraints and the available resources change over time in multi-tenant environments, it is necessary to design an approach which provides a flexible configuration space. This paper presents an adaptive serving approach for the efficient deployment of MoE models, capitalizing on partial quantization of the experts. By dynamically determining the number of quantized experts and their distribution across CPU and GPU, our approach explores the Pareto frontier and offers a fine-grained range of configurations for tuning throughput and model quality. Our evaluation on an NVIDIA A100 GPU using a Mixtral 8x7B MoE model for three language modelling benchmarks demonstrates that the throughput of token generation can be adjusted from 0.63 to 13.00 token per second. This enhancement comes with a marginal perplexity increase of 3.81 to 4.00, 13.59 to 14.17, and 7.24 to 7.40 for WikiText2, PTB, and C4 datasets respectively under maximum quantization. These results highlight the practical applicability of our approach in dynamic and accuracy-sensitive applications where both memory usage and output quality are important.