Stable Update of Regression Trees
This work addresses the need for stable and explainable model updates in applications where consistency matters, though it is incremental as it builds on existing regression tree methods.
The paper tackles the problem of updating regression trees to balance predictive performance with stability, ensuring predictions do not change too much when new data is added. The results show that the proposed regularization method improves stability while maintaining or enhancing predictive accuracy.
Updating machine learning models with new information usually improves their predictive performance, yet, in many applications, it is also desirable to avoid changing the model predictions too much. This property is called stability. In most cases when stability matters, so does explainability. We therefore focus on the stability of an inherently explainable machine learning method, namely regression trees. We aim to use the notion of empirical stability and design algorithms for updating regression trees that provide a way to balance between predictability and empirical stability. To achieve this, we propose a regularization method, where data points are weighted based on the uncertainty in the initial model. The balance between predictability and empirical stability can be adjusted through hyperparameters. This regularization method is evaluated in terms of loss and stability and assessed on a broad range of data characteristics. The results show that the proposed update method improves stability while achieving similar or better predictive performance. This shows that it is possible to achieve both predictive and stable results when updating regression trees.