MP-Boost: Minipatch Boosting via Adaptive Feature and Observation Sampling
This work addresses the need for efficient and interpretable boosting algorithms in machine learning, though it is incremental as it builds on existing AdaBoost concepts.
The authors tackled the problem of improving computational speed and interpretability in boosting methods, achieving comparable accuracy to AdaBoost and gradient boosting while being faster and more interpretable through adaptive minipatch sampling.
Boosting methods are among the best general-purpose and off-the-shelf machine learning approaches, gaining widespread popularity. In this paper, we seek to develop a boosting method that yields comparable accuracy to popular AdaBoost and gradient boosting methods, yet is faster computationally and whose solution is more interpretable. We achieve this by developing MP-Boost, an algorithm loosely based on AdaBoost that learns by adaptively selecting small subsets of instances and features, or what we term minipatches (MP), at each iteration. By sequentially learning on tiny subsets of the data, our approach is computationally faster than other classic boosting algorithms. Also as it progresses, MP-Boost adaptively learns a probability distribution on the features and instances that upweight the most important features and challenging instances, hence adaptively selecting the most relevant minipatches for learning. These learned probability distributions also aid in interpretation of our method. We empirically demonstrate the interpretability, comparative accuracy, and computational time of our approach on a variety of binary classification tasks.