Eliminating Multicollinearity Issues in Neural Network Ensembles: Incremental, Negatively Correlated, Optimal Convex Blending
This addresses a known bottleneck in ensemble methods for machine learning practitioners, offering an incremental improvement.
The paper tackles the multicollinearity problem in neural network ensembles by introducing an incremental algorithm that optimally blends regressors under a convexity constraint, inducing negative correlations to eliminate redundancy and improve accuracy and robustness.
Given a {features, target} dataset, we introduce an incremental algorithm that constructs an aggregate regressor, using an ensemble of neural networks. It is well known that ensemble methods suffer from the multicollinearity issue, which is the manifestation of redundancy arising mainly due to the common training-dataset. In the present incremental approach, at each stage we optimally blend the aggregate regressor with a newly trained neural network under a convexity constraint which, if necessary, induces negative correlations. Under this framework, collinearity issues do not arise at all, rendering so the method both accurate and robust.