Branches: Efficiently Seeking Optimal Sparse Decision Trees with AO*
This addresses the inefficiency in searching for optimal decision trees at high depths for interpretable machine learning, offering a novel solution with theoretical and practical improvements.
The paper tackles the optimization challenge in learning sparse decision trees by proposing Branches, an AO*-type algorithm that formulates the problem as an AND/OR graph search, proving optimality and complexity guarantees and showing it is more efficient than state-of-the-art methods in experiments.
Decision Tree (DT) Learning is a fundamental problem in Interpretable Machine Learning, yet it poses a formidable optimisation challenge. Practical algorithms have recently emerged, primarily leveraging Dynamic Programming and Branch & Bound. However, most of these approaches rely on a Depth-First-Search strategy, which is inefficient when searching for DTs at high depths and requires the definition of a maximum depth hyperparameter. Best-First-Search was also employed by other methods to circumvent these issues. The downside of this strategy is its higher memory consumption, as such, it has to be designed in a fully efficient manner that takes full advantage of the problem's structure. We formulate the problem within an AND/OR graph search framework and we solve it with a novel AO*-type algorithm called Branches. We prove both optimality and complexity guarantees for Branches and we show that it is more efficient than the state of the art theoretically and on a variety of experiments. Furthermore, Branches supports non-binary features unlike the other methods, we show that this property can further induce larger gains in computational efficiency.