Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE
This work addresses the challenge of enhancing reasoning diversity without destabilizing outputs in fine-grained MoE models, which is incremental as it builds on existing test-time scaling methods by leveraging routing patterns.
The paper tackles the problem of improving test-time scaling in fine-grained Mixture of Experts (MoE) models by addressing the trade-off between diversity and stability in token-level sampling. It introduces Expert-Sample, a training-free method that preserves high-confidence expert selections while injecting stochasticity into low-confidence ones, resulting in improved pass@n and accuracy, such as raising pass@32 from 85.4% to 91.9% and accuracy from 59.1% to 62.6% on GPQA-Diamond with Qwen3-30B-A3B-Instruct.
Test-time scaling improves LLM performance by generating multiple candidate solutions, yet token-level sampling requires temperature tuning that trades off diversity against stability. Fine-grained MoE, featuring hundreds of well-trained experts per layer and multi-expert activation per token, offers an unexplored alternative through its rich routing space. We empirically characterize fine-grained MoE routing and uncover an informative pattern: router scores exhibit a certain head of high-confidence experts followed by an uncertain tail of low-confidence candidates. While single-run greedy accuracy remains stable when fewer experts are activated, multi-sample pass@n degrades significantly-suggesting that the certain head governs core reasoning capability while the uncertain tail correlates with reasoning diversity. Motivated by these findings, we propose Expert-Sample, a training-free method that preserves high-confidence selections while injecting controlled stochasticity into the uncertain tail, enabling diverse generation without destabilizing outputs. Evaluated on multiple fine-grained MoE models across math, knowledge reasoning, and code tasks, Expert-Sample consistently improves pass@n and verification-based accuracy. On Qwen3-30B-A3B-Instruct evaluated on GPQA-Diamond with 32 parallel samples, pass@32 rises from 85.4% to 91.9%, and accuracy improves from 59.1% to 62.6% with Best-of-N verification.