Scholar Feed

Mixture-of-experts routing

Upcycling

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

Superseded baseline#12 of 1,370 most-superseded · first seen Dec 9, 2022

Superseded — cited as a baseline and beaten by newer methods

3 papers critique it · 2 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Upcycling as a baseline.

its benefits typically emerge only after extensive training, often exceeding practical instruction-tuning budgets
— Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
“MoE models initialized with Upcycling tend to have a much slower convergence, leading to suboptimal performance when trained for longer durations.”
— Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
“However, initializing all experts with identical weights alongside a randomly initialized router inherently introduces expert symmetry; consequently, the model lacks a meaningful basis for early specialization.”
— Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling

Beaten on benchmarks

Head-to-head results where a newer method reports beating Upcycling. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.