Complexity-Optimized Sparse Bayesian Learning for Scalable Classification Tasks
This work addresses a computational bottleneck for researchers and practitioners using SBL in high-dimensional or large-scale classification problems, representing an incremental improvement.
The paper tackles the scalability issue of Sparse Bayesian Learning (SBL), which has high computational complexity due to matrix inversion, by proposing a diagonal Quasi-Newton method (DQN-SBL) that reduces complexity from O(M^3) to O(M). Experimental results show that DQN-SBL achieves competitive generalization with sparse models and scales well to large-scale classification tasks.
Sparse Bayesian Learning (SBL) constructs an extremely sparse probabilistic model with very competitive generalization. However, SBL needs to invert a big covariance matrix with complexity $O(M^3)$ (M: feature size) for updating the regularization priors, making it difficult for problems with high dimensional feature space or large data size. As it may easily suffer from the memory overflow issue in such problems. This paper addresses this issue with a newly proposed diagonal Quasi-Newton (DQN) method for SBL called DQN-SBL where the inversion of big covariance matrix is ignored so that the complexity is reduced to $O(M)$. The DQN-SBL is thoroughly evaluated for non linear and linear classifications with various benchmarks of different sizes. Experimental results verify that DQN-SBL receives competitive generalization with a very sparse model and scales well to large-scale problems.