Asymptotic Analysis of an Ensemble of Randomly Projected Linear Discriminants
This work provides theoretical insights for researchers in bioinformatics, chemometrics, and face recognition dealing with high-dimensional data, but it is incremental as it builds on existing ensemble methods.
The authors tackled the problem of understanding the behavior of an ensemble of randomly projected linear discriminant classifiers for high-dimensional, small-sample data by performing asymptotic analysis, deriving misclassification probabilities that show the ensemble regularizes the covariance matrix and developing a consistent estimator for tuning parameters.
Datasets from the fields of bioinformatics, chemometrics, and face recognition are typically characterized by small samples of high-dimensional data. Among the many variants of linear discriminant analysis that have been proposed in order to rectify the issues associated with classification in such a setting, the classifier in [1], composed of an ensemble of randomly projected linear discriminants, seems especially promising; it is computationally efficient and, with the optimal projection dimension parameter setting, is competitive with the state-of-the-art. In this work, we seek to further understand the behavior of this classifier through asymptotic analysis. Under the assumption of a growth regime in which the dataset and projection dimensions grow at constant rates to each other, we use random matrix theory to derive asymptotic misclassification probabilities showing the effect of the ensemble as a regularization of the data sample covariance matrix. The asymptotic errors further help to identify situations in which the ensemble offers a performance advantage. We also develop a consistent estimator of the misclassification probability as an alternative to the computationally-costly cross-validation estimator, which is conventionally used for parameter tuning. Finally, we demonstrate the use of our estimator for tuning the projection dimension on both real and synthetic data.