Scholar Feed

Mixture-of-experts routing

Fiddler

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

Superseded baseline#8 of 1,370 most-superseded · first seen Feb 10, 2024

Superseded — cited as a baseline and beaten by newer methods

7 papers critique it · 2 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Fiddler as a baseline.

Fiddler fiddler dynamically places experts across CPU and GPU, yet lacks precise scheduling for hot experts (those processing more tokens).
— PreScope: Unleashing the Power of Prefetching for Resource-Constrained MoE Inference
“For example, Fiddler uses fixed mapping based on expert activation frequency for CPU-GPU scheduling, which fails to adapt to changing loads.”
— HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
“Additionally, approaches such as Fiddler leverage CPU for additional compute power, but do not fully explore the characteristics of MoE models.”
— MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
“Fiddler reduces PCIe traffic by executing some expert computation on the CPU, but its gains are contingent on CPU capability and diminish as per-expert token counts grow, where CPU execution becomes slow and weight transfers to GPU become preferable.”
— MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
“While Fiddler and DAOP aim for DRAM-offloading-based inference, their CPU-based computation cannot be fully utilized due to memory bottlenecks.”
— FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
“we use Fiddler as a CPU computation baseline, where expert placements remain static during decoding”
— TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload
“Heterogeneous strategies like Fiddler offload certain computations to the CPU but encounter compute-bound bottlenecks during dequantization, leading to latency penalties that outweigh transmission savings”
— DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge

Beaten on benchmarks

Head-to-head results where a newer method reports beating Fiddler. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.