Is Mixtral superseded?

Mixtral (Mixture-of-experts routing): superseded — cited as a baseline and beaten by newer methods. 1 paper(s) critique it, 2 beat it on benchmarks — #10 of 1370 most-superseded. Sub-problem: cluster led by Mixtral. Newer alternatives in the same sub-problem include PADD, MESA, PR2, RouteHijack, MASCing.

Mixture-of-experts routing

Mixtral

Mixtral of Experts

Superseded baseline#10 of 1,370 most-superseded · first seen Jan 8, 2024

Superseded — cited as a baseline and beaten by newer methods

1 papers critique it · 2 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Mixtral as a baseline.

They show that the choice of experts seems to be influenced more by syntax than by domain, particularly in the first and last layers

— Steering MoE LLMs via Expert (De)Activation

Beaten on benchmarks

Head-to-head results where a newer method reports beating Mixtral. Values are copied from the source paper's tables — verify against the cited paper.

AdaMoE beats Mixtral
1.66 vs 2.00
Average Load · [Fine-tuned Mixtral-8x7B (top-2 routing)]
AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models
WDMoE beats Mixtral
37.4 vs 35.2
MBPP · [all benchmarks]
WDMoE: Wireless Distributed Mixture of Experts for Large Language Models

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.