Is Branch-Train-Merge superseded?

Branch-Train-Merge (Mixture-of-experts routing): superseded — cited as a baseline and beaten by newer methods. 3 paper(s) critique it, 0 beat it on benchmarks — #65 of 1370 most-superseded. Sub-problem: cluster led by BTX. Newer alternatives in the same sub-problem include MetaMoE, BAR, BERT-MoE Framework, null experts within token-choice MoE, MixtureKit.

Method Drift›Mixture-of-experts routing

Superseded baseline#65 of 1,370 most-superseded

Branch-Train-Merge

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Mixture-of-experts routing · first seen Aug 5, 2022

superseded — cited as a baseline and beaten by newer methods

3 papers critique it · 0 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Branch-Train-Merge as a baseline.

“does not yield a single unified model (hindering downstream SFT/RLHF and incurring inference overhead)”
— MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
“While this approach makes training more efficient, its main drawback is the lack of a unified single model making it impossible to do further supervised finetuning (SFT) or reinforcement learning from human feedback (RLHF) finetuning”
— Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
“Despite its strong performance, this architecture limited the ability to further fine-tune the individual experts' components within the unified structure.”
— MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models

Newer alternatives

Recent methods in the same sub-problem, not yet superseded in the knowledge base.