Neuronal Attention Circuit (NAC) for Representation Learning
This addresses the problem of efficient and interpretable continuous-time attention for researchers in machine learning and applications like autonomous systems, though it is incremental as it builds on existing attention and neuronal circuit methods.
The paper tackled the limitation of discrete attention in continuous-time modeling by introducing Neuronal Attention Circuit (NAC), a biologically inspired mechanism that reformulates attention as a linear ODE with sparse gates, and it matched or outperformed baselines in accuracy across domains like time-series classification and autonomous driving, with intermediate runtime and memory usage.
Attention improves representation learning over RNNs, but its discrete nature limits continuous-time (CT) modeling. We introduce Neuronal Attention Circuit (NAC), a novel, biologically inspired CT-Attention mechanism that reformulates attention logit computation as the solution to a linear first-order ODE with nonlinear interlinked gates derived from repurposing C.elegans Neuronal Circuit Policies (NCPs) wiring. NAC replaces dense projections with sparse sensory gates for key-query projections and a sparse backbone network with two heads for computing content-target and learnable time-constant gates, enabling efficient adaptive dynamics. To improve efficiency and memory consumption, we implemented an adaptable subquadratic sparse Top-K pairwise concatenation mechanism that selectively curates key-query interactions. We provide rigorous theoretical guarantees, including state stability and bounded approximation errors. Empirically, we implemented NAC in diverse domains, including irregular time-series classification, lane-keeping for autonomous vehicles, and industrial prognostics. We observed that NAC matches or outperforms competing baselines in accuracy and occupies an intermediate position in runtime and memory consumption compared with several CT state-of-the-art baselines, while being interpretable at the neuron cell level.