Probabilistic Classification Vector Machine for Multi-Class Classification
This work addresses the problem of multi-class classification for machine learning practitioners by extending a sparse Bayesian method, though it is incremental as it builds on existing PCVM frameworks.
The authors tackled the limitation of the probabilistic classification vector machine (PCVM) to binary classification by proposing a multi-class extension (mPCVM) with two learning algorithms, achieving superior performance on synthetic and benchmark datasets, especially for problems with many classes.
The probabilistic classification vector machine (PCVM) synthesizes the advantages of both the support vector machine and the relevant vector machine, delivering a sparse Bayesian solution to classification problems. However, the PCVM is currently only applicable to binary cases. Extending the PCVM to multi-class cases via heuristic voting strategies such as one-vs-rest or one-vs-one often results in a dilemma where classifiers make contradictory predictions, and those strategies might lose the benefits of probabilistic outputs. To overcome this problem, we extend the PCVM and propose a multi-class probabilistic classification vector machine (mPCVM). Two learning algorithms, i.e., one top-down algorithm and one bottom-up algorithm, have been implemented in the mPCVM. The top-down algorithm obtains the maximum a posteriori (MAP) point estimates of the parameters based on an expectation-maximization algorithm, and the bottom-up algorithm is an incremental paradigm by maximizing the marginal likelihood. The superior performance of the mPCVMs, especially when the investigated problem has a large number of classes, is extensively evaluated on synthetic and benchmark data sets.