Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning
This work addresses the problem of inefficient and non-interpretable conditional computation for AI researchers and practitioners, offering a framework for specialized reasoning in low-resource regimes, though it is incremental in improving MoE architectures.
The paper tackled the limitation of static computation pathways in Mixture-of-Experts models by proposing Hecto, a lightweight architecture with heterogeneous experts (GRU for temporal reasoning and FFNN for static abstraction) under sparse gating, which matched or closely trailed homogeneous baselines on reasoning benchmarks like AG News and HotpotQA while achieving clear expert specialization.
Mixture-of-Experts (MoE) models enable conditional computation by routing inputs to specialized experts, but these experts rely on identical inductive biases, thus limiting representational diversity. This static computation pathway is inefficient for inputs that require different types of reasoning and limits specialization and interpretability. We propose Hecto, a lightweight MoE architecture that leverages architectural heterogeneity by combining a GRU expert for temporal reasoning and an FFNN expert for static abstraction under a sparse Top-1 gating mechanism. Evaluated on three reasoning benchmarks (AG News, SST-2, HotpotQA) and a regression task (STS-B), Hecto matches or closely trails homogeneous baselines in performance despite receiving isolated input representations, while achieving clear expert specialization, with each expert aligning to distinct reasoning types (temporal vs static). At larger batch sizes, Hecto exhibits improved performance, benefiting from relaxed computational constraints that allow its heterogeneous architecture to optimize more effectively. Ablation results isolate architectural diversity as the source of Hecto's stability and interpretability across diverse reasoning tasks. Overall, Hecto establishes itself as a new benchmark for conditional computation, offering a principled framework for specialized reasoning in low-resource regimes with its model strength derived from principled specialization.