Bags of Projected Nearest Neighbours: Competitors to Random Forests?
This work addresses the need for more effective ensemble methods in machine learning, particularly for classification tasks, though it appears incremental as it builds on existing nearest neighbor and bagging techniques.
The authors tackled the problem of improving nearest neighbor classifiers by introducing an adaptive k-nearest neighbors method that uses discriminant subspaces to enhance class discrimination, and found that bagging this approach yields strong performance comparable to Random Forests and other benchmarks.
In this paper we introduce a simple and intuitive adaptive k nearest neighbours classifier, and explore its utility within the context of bootstrap aggregating ("bagging"). The approach is based on finding discriminant subspaces which are computationally efficient to compute, and are motivated by enhancing the discrimination of classes through nearest neighbour classifiers. This adaptiveness promotes diversity of the individual classifiers fit across different bootstrap samples, and so further leverages the variance reducing effect of bagging. Extensive experimental results are presented documenting the strong performance of the proposed approach in comparison with Random Forest classifiers, as well as other nearest neighbours based ensembles from the literature, plus other relevant benchmarks. Code to implement the proposed approach is available in the form of an R package from https://github.com/DavidHofmeyr/BOPNN.