Conditional Activation for Diverse Neurons in Heterogeneous Networks
This work addresses the challenge of efficiently modeling diverse neuron behaviors in neural networks, which is incremental as it builds on existing heterogeneous network approaches.
The paper tackles the problem of modeling diverse neuron behavior by introducing conditional activation, where a neuron's activation function is dynamically modified by a control signal. The method, applied to heterogeneous MLPs, demonstrates simultaneous improvements in learning speed and performance across various network configurations, with significant reductions in memory for storing parameters.
In this paper, we propose a new scheme for modelling the diverse behavior of neurons. We introduce the conditional activation, in which a neurons activation function is dynamically modified by a control signal. We apply this method to recreate behavior of special neurons existing in the human auditory and visual system. A heterogeneous multilayered perceptron (MLP) incorporating the developed models demonstrates simultaneous improvement in learning speed and performance across a various number of hidden units and layers, compared to a homogeneous network composed of the conventional neuron model. For similar performance, the proposed model lowers the memory for storing network parameters significantly.