AGBoost: Attention-based Modification of Gradient Boosting Machine
This work addresses regression tasks for machine learning practitioners, offering an incremental improvement by integrating attention mechanisms into gradient boosting.
The authors tackled regression problems by proposing AGBoost, an attention-based modification of gradient boosting machines that assigns trainable attention weights to iterations, resulting in improved performance as shown in numerical experiments with various datasets.
A new attention-based model for the gradient boosting machine (GBM) called AGBoost (the attention-based gradient boosting) is proposed for solving regression problems. The main idea behind the proposed AGBoost model is to assign attention weights with trainable parameters to iterations of GBM under condition that decision trees are base learners in GBM. Attention weights are determined by applying properties of decision trees and by using the Huber's contamination model which provides an interesting linear dependence between trainable parameters of the attention and the attention weights. This peculiarity allows us to train the attention weights by solving the standard quadratic optimization problem with linear constraints. The attention weights also depend on the discount factor as a tuning parameter, which determines how much the impact of the weight is decreased with the number of iterations. Numerical experiments performed for two types of base learners, original decision trees and extremely randomized trees with various regression datasets illustrate the proposed model.