Neural Inhibition Improves Dynamic Routing and Mixture of Experts
This work addresses a bottleneck in dynamic routing and Mixture-of-Experts models for deep learning practitioners, presenting an incremental improvement.
The paper tackles the problem of improving dynamic routing in deep learning models by introducing neural inhibition to enhance specialization among expert pathways, resulting in significant performance boosts on general tasks.
To be effective, efficient, and diverse, deep learning models need to dynamically choose its architecture based on signals from a population of neurons. We hypothesize dynamic routing models can be improved with neural inhibition in those neural populations. This means signals commonly shared among the various modes of data statistics can be inhibited so that the routing model can choose a specialized expert path for each data sample. Only through inhibition is the routing mechanism able to effectively select neural pathways. We believe this is an under-studied and under-verified implementation methodology for Mixture-of-Experts, dynamic routing, and transformer language models. We provide experimental evidence that the neural inhibition algorithm significantly boosts the performance of general tasks and motivates more effort to be invested in this research direction.