Learning Representations for Axis-Aligned Decision Forests through Input Perturbation
This work addresses a limitation for users of decision forests in applications requiring representation learning, such as transfer learning, but it is incremental as it builds on existing methods by integrating neural networks without structural changes to forests.
The paper tackles the problem of enabling axis-aligned decision forests to learn effective representations from raw structured data like text, which they traditionally cannot handle, by proposing a framework that combines a decision forest with a deep neural network and uses input perturbation to approximate gradients, resulting in demonstrated feasibility and effectiveness on synthetic and benchmark datasets.
Axis-aligned decision forests have long been the leading class of machine learning algorithms for modeling tabular data. In many applications of machine learning such as learning-to-rank, decision forests deliver remarkable performance. They also possess other coveted characteristics such as interpretability. Despite their widespread use and rich history, decision forests to date fail to consume raw structured data such as text, or learn effective representations for them, a factor behind the success of deep neural networks in recent years. While there exist methods that construct smoothed decision forests to achieve representation learning, the resulting models are decision forests in name only: They are no longer axis-aligned, use stochastic decisions, or are not interpretable. Furthermore, none of the existing methods are appropriate for problems that require a Transfer Learning treatment. In this work, we present a novel but intuitive proposal to achieve representation learning for decision forests without imposing new restrictions or necessitating structural changes. Our model is simply a decision forest, possibly trained using any forest learning algorithm, atop a deep neural network. By approximating the gradients of the decision forest through input perturbation, a purely analytical procedure, the decision forest directs the neural network to learn or fine-tune representations. Our framework has the advantage that it is applicable to any arbitrary decision forest and that it allows the use of arbitrary deep neural networks for representation learning. We demonstrate the feasibility and effectiveness of our proposal through experiments on synthetic and benchmark classification datasets.