Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
This addresses the problem of efficient inference for large language models in settings with limited GPU memory, representing an incremental improvement over existing systems.
The paper tackles the challenge of running large Mixture-of-Experts models in resource-constrained environments by proposing Fiddler, a system that orchestrates CPU and GPU resources for efficient inference, achieving speedups of up to 11.57 times in beam search compared to baselines.
Large Language Models (LLMs) with the Mixture-of-Experts (MoE) architectures have shown promising performance on various tasks. However, due to the huge model sizes, running them in resource-constrained environments where the GPU memory is not abundant is challenging. Some existing systems propose to use CPU resources to solve that, but they either suffer from the significant overhead of frequently moving data between CPU and GPU, or fail to consider distinct characteristics of CPUs and GPUs. This paper proposes Fiddler, a resource-efficient inference system for MoE models with limited GPU resources. Fiddler strategically utilizes CPU and GPU resources by determining the optimal execution strategy. Our evaluation shows that, unlike state-of-the-art systems that optimize for specific scenarios such as single batch inference or long prefill, Fiddler performs better in all scenarios. Compared against different baselines, Fiddler achieves 1.26 times speed up in single batch inference, 1.30 times in long prefill processing, and 11.57 times in beam search inference. The code of Fiddler is publicly available at https://github.com/efeslab/fiddler.