Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations
For practitioners using MoE models, CoR offers a practical, training-free method to reduce hallucinations, particularly for long-tail knowledge, without additional computational cost.
The paper identifies that hallucinations in sparse Mixture-of-Experts (MoE) models stem from static Top-k routing, which under-prioritizes specialist experts for long-tail knowledge. They propose Counterfactual Routing (CoR), a training-free inference framework that improves factual accuracy by 3.1% on average without increasing inference budget.
Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top-$k$ routing: routers tend to favor high-frequency patterns over rare factual associations. Consequently, ``specialist experts'' possessing critical long-tail knowledge are often assigned low gating scores and remain ``dormant'' -- under-prioritized for specific tokens despite their proven causal importance on other inputs. To address this, we propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken these dormant experts. CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources from syntax-dominant to knowledge-intensive layers while maintaining a constant total activation count, effectively retrieving causally decisive experts via virtual ablation. Extensive experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1\% on average without increasing the inference budget, establishing a superior Pareto frontier compared to static scaling strategies.