Is Branch-Train-Merge superseded?

Branch-Train-Merge (Mixture-of-experts routing): cited as a baseline — critiqued by newer work, not yet beaten on a benchmark here. 3 paper(s) critique it, 0 beat it on benchmarks — #24 of 1370 most-superseded. Sub-problem: cluster led by Upcycling. Newer alternatives in the same sub-problem include MetaMoE, BAR, BERT-MoE Framework, null experts within token-choice MoE, MixtureKit.

Mixture-of-experts routing

Branch-Train-Merge

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Superseded baseline#24 of 1,370 most-superseded · first seen Aug 5, 2022

Cited as a baseline — critiqued by newer work, not yet beaten on a benchmark here

3 papers critique it · 0 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Branch-Train-Merge as a baseline.

does not yield a single unified model (hindering downstream SFT/RLHF and incurring inference overhead)

— MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification

“While this approach makes training more efficient, its main drawback is the lack of a unified single model making it impossible to do further supervised finetuning (SFT) or reinforcement learning from human feedback (RLHF) finetuning”

— Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

“Despite its strong performance, this architecture limited the ability to further fine-tune the individual experts' components within the unified structure.”

— MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.