Mixtral-Offloading

“Most inference engines (e.g., DeepSpeed, Mixtral-Offloading) use prediction-based methods to manage their expert cache within GPUs. These methods primarily analyze the execution order of experts in the computational graph (e.g., prioritizing experts in the next immediate layer). While prediction-based methods work well for fully activated dense models, they fail to account for the sparse activation of experts and assume all required experts must be fetched into GPUs, leading to substantial I/O bottlenecks on the PCIe bus.”

“However, I/O slowdowns still hinder improvements in end-to-end inference latency. Their aggressive mixed-precision quantization also trades model quality for memory efficiency.”

“existing expert offloading solutions struggle to effectively balance the latency-memory trade-off in serving. These approaches either suffer from high inference latency”

“Systems like FlexGen~flexgen, DeepSpeed-Inference~ds-infer, Mixtral-Offloading~mixtral-offload, and MoE-Lightning~moe-lightning often operate with batch sizes 40–1000× smaller than what is needed to fully utilize a GPU during LLM decode, significantly reducing throughput compared to the prefill phase.”

“expert migration at every denoising step is prohibitively expensive, as a single dLLM step activates a larger, more diverse set of experts than an AR step, thus creating massive CPU-GPU I/O traffic.”