Orthogonal Stochastic Configuration Networks with Adaptive Construction Parameter for Data Analytics
This work addresses the issue of non-compact network structures in randomized learner models for data analytics, though it is incremental as it builds on existing SCNs with orthogonalization techniques.
The paper tackled the problem of redundant and low-quality nodes in Stochastic Configuration Networks (SCNs) by proposing Orthogonal SCN (OSCN), which uses Gram-Schmidt orthogonalization to filter nodes and reduce network structure, resulting in improved generalization and computational efficiency as validated on regression and classification datasets.
As a randomized learner model, SCNs are remarkable that the random weights and biases are assigned employing a supervisory mechanism to ensure universal approximation and fast learning. However, the randomness makes SCNs more likely to generate approximate linear correlative nodes that are redundant and low quality, thereby resulting in non-compact network structure. In the light of a fundamental principle in machine learning, that is, a model with fewer parameters holds improved generalization. This paper proposes orthogonal SCN, termed OSCN, to filtrate out the low-quality hidden nodes for network structure reduction by incorporating Gram-Schmidt orthogonalization technology. The universal approximation property of OSCN and an adaptive setting for the key construction parameters have been presented in details. In addition, an incremental updating scheme is developed to dynamically determine the output weights, contributing to improved computational efficiency. Finally, experimental results on two numerical examples and several real-world regression and classification datasets substantiate the effectiveness and feasibility of the proposed approach.