Learning optimal Bayesian prior probabilities from data
This work offers a significant improvement in Bayesian inference for practitioners who use Naïve Bayes text classification, particularly for tasks like document categorization.
This study challenges the use of noninformative uniform priors in Bayesian inference, proposing a method to learn optimal priors from data by maximizing a target function. Applied to a Naïve Bayes text classification task, the proposed method achieved a performance improvement of up to 443% (mean 193%) over baseline models using Bayes-Laplace priors.
Noninformative uniform priors are staples of Bayesian inference, especially in Bayesian machine learning. This study challenges the assumption that they are optimal and their use in Bayesian inference yields optimal outcomes. Instead of using arbitrary noninformative uniform priors, we propose a machine learning based alternative method, learning optimal priors from data by maximizing a target function of interest. Applying naïve Bayes text classification methodology and a search algorithm developed for this study, our system learned priors from data using the positive predictive value metric as the target function. The task was to find Wikipedia articles that had not (but should have) been categorized under certain Wikipedia categories. We conducted five sets of experiments using separate Wikipedia categories. While the baseline models used the popular Bayes-Laplace priors, the study models learned the optimal priors for each set of experiments separately before using them. The results showed that the study models consistently outperformed the baseline models with a wide margin of statistical significance (p < 0.001). The measured performance improvement of the study model over the baseline was as high as 443% with the mean value of 193% over five Wikipedia categories.