Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
This addresses the challenge of efficient reasoning enhancement in MoE LLMs for AI applications, representing an incremental improvement by leveraging architectural flexibility.
The paper tackles the problem of enhancing reasoning in Mixture-of-Experts LLMs by proposing Dynamic Experts Search (DES), a test-time scaling strategy that controls expert activation to generate diverse reasoning trajectories, resulting in improved accuracy and stability without additional cost across math, code, and knowledge benchmarks.
Test-Time Scaling (TTS) enhances the reasoning ability of large language models (LLMs) by allocating additional computation during inference. However, existing approaches primarily rely on output-level sampling while overlooking the role of model architecture. In mainstream Mixture-of-Experts (MoE) LLMs, we observe that varying the number of activated experts yields complementary solution sets with stable accuracy, revealing a new and underexplored source of diversity. Motivated by this observation, we propose Dynamic Experts Search (DES), a TTS strategy that elevates expert activation into a controllable dimension of the search space. DES integrates two key components: (1) Dynamic MoE, which enables direct control of expert counts during inference to generate diverse reasoning trajectories without additional cost; and (2) Expert Configuration Inheritance, which preserves consistent expert counts within a reasoning path while varying them across runs, thereby balancing stability and diversity throughout the search. Extensive experiments across MoE architectures, verifiers and reasoning benchmarks (i.e., math, code and knowledge) demonstrate that DES reliably outperforms TTS baselines, enhancing accuracy and stability without additional cost. These results highlight DES as a practical and scalable form of architecture-aware TTS, illustrating how structural flexibility in modern LLMs can advance reasoning.