Heterogeneous Multilayer Generalized Operational Perceptron
This work addresses the problem of designing more flexible neural network architectures for machine learning practitioners, but it appears incremental as it builds on prior Generalized Operational Perceptron models.
The paper tackles the limitation of traditional Multilayer Perceptrons by proposing an algorithm to learn compact, fully heterogeneous multilayer networks where each neuron has distinct characteristics, and it outperforms other methods in classification experiments.
The traditional Multilayer Perceptron (MLP) using McCulloch-Pitts neuron model is inherently limited to a set of neuronal activities, i.e., linear weighted sum followed by nonlinear thresholding step. Previously, Generalized Operational Perceptron (GOP) was proposed to extend conventional perceptron model by defining a diverse set of neuronal activities to imitate a generalized model of biological neurons. Together with GOP, Progressive Operational Perceptron (POP) algorithm was proposed to optimize a pre-defined template of multiple homogeneous layers in a layerwise manner. In this paper, we propose an efficient algorithm to learn a compact, fully heterogeneous multilayer network that allows each individual neuron, regardless of the layer, to have distinct characteristics. Based on the complexity of the problem, the proposed algorithm operates in a progressive manner on a neuronal level, searching for a compact topology, not only in terms of depth but also width, i.e., the number of neurons in each layer. The proposed algorithm is shown to outperform other related learning methods in extensive experiments on several classification problems.