Tutel

“existing MoE training systems, such as DeepSpeed-MoE~deepspeed-moe, DeepSpeed-TED~deepspeed-ted, and Tutel~hwang2023tutel, do not effectively address this shifted bottleneck, causing a memory explosion”

“it still struggles with load imbalance due to static expert assignment”

“their current capabilities are limited to manual configuration of the pipeline degree or heuristic search methods within a constrained search space”

“Tutel~tutel2022 provides efficient MoE dispatch and combine kernels with dynamic top-K routing and adaptive parallelism switching, but focuses on the dispatch kernel rather than end-to-end training strategy selection and does not cover attention-layer parallelization.”

“Tutel tutel and Comet zhang2025comet propose operator-level chunking, which partitions FFN computations into fine-grained tiles and partially overlaps dispatch/combine with the MoE layer through tile-level pipelining. However, as shown in Fig.~fig:motivationchunk, the overlap window is bounded by FFN computation, leaving residual communication exposed.”

“Tutel and OpenMoE+t5x are primarily designed for large-scale pretraining on hundreds of GPUs, restricting accessibility for groups with limited resources.”