Opening the Black Box: Interpretable LLMs via Semantic Resonance Architecture
This addresses the need for more transparent and controllable language models for AI researchers and practitioners, representing a novel method rather than an incremental improvement.
The researchers tackled the problem of interpretability in large language models by developing the Semantic Resonance Architecture (SRA), which replaces opaque gating functions with cosine similarity-based routing to semantic anchors, achieving a validation perplexity of 13.41 on WikiText-103 and reducing dead experts from 14.8% to 1.0% compared to standard MoE models.
Large language models (LLMs) achieve remarkable performance but remain difficult to interpret. Mixture-of-Experts (MoE) models improve efficiency through sparse activation, yet typically rely on opaque, learned gating functions. While similarity-based routing (Cosine Routers) has been explored for training stabilization, its potential for inherent interpretability remains largely untapped. We introduce the Semantic Resonance Architecture (SRA), an MoE approach designed to ensure that routing decisions are inherently interpretable. SRA replaces learned gating with a Chamber of Semantic Resonance (CSR) module, which routes tokens based on cosine similarity with trainable semantic anchors. We also introduce a novel Dispersion Loss that encourages orthogonality among anchors to enforce diverse specialization. Experiments on WikiText-103 demonstrate that SRA achieves a validation perplexity of 13.41, outperforming both a dense baseline (14.13) and a Standard MoE baseline (13.53) under matched active parameter constraints (29.0M). Crucially, SRA exhibits superior expert utilization (1.0% dead experts vs. 14.8% in the Standard MoE) and develops distinct, semantically coherent specialization patterns, unlike the noisy specialization observed in standard MoEs. This work establishes semantic routing as a robust methodology for building more transparent and controllable language models.