Optimal Projections for Classification with Naive Bayes
This work addresses the need for more effective probabilistic classification methods in machine learning, offering a novel approach that enhances discriminatory power while providing dimension reduction and visualization benefits.
The authors tackled the problem of improving Naive Bayes classification by finding an optimal linear projection for factorizing class conditional densities, which substantially outperformed other probabilistic discriminant analysis models and was highly competitive with Support Vector Machines on 162 benchmark datasets.
In the Naive Bayes classification model the class conditional densities are estimated as the products of their marginal densities along the cardinal basis directions. We study the problem of obtaining an alternative basis for this factorisation with the objective of enhancing the discriminatory power of the associated classification model. We formulate the problem as a projection pursuit to find the optimal linear projection on which to perform classification. Optimality is determined based on the multinomial likelihood within which probabilities are estimated using the Naive Bayes factorisation of the projected data. Projection pursuit offers the added benefits of dimension reduction and visualisation. We discuss an intuitive connection with class conditional independent components analysis, and show how this is realised visually in practical applications. The performance of the resulting classification models is investigated using a large collection of (162) publicly available benchmark data sets and in comparison with relevant alternatives. We find that the proposed approach substantially outperforms other popular probabilistic discriminant analysis models and is highly competitive with Support Vector Machines. Code to implement the proposed approach, in the form of an R package, is available from https://github.com/DavidHofmeyr/OPNB