Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning
This work addresses the challenge of improving tabular prediction tasks for data scientists and ML practitioners by offering a novel, iterative feature engineering approach, though it is incremental in combining existing LLM and tree-based techniques.
The paper tackles the problem of automated feature engineering for tabular data by proposing OCTree, a framework that uses large language models (LLMs) with decision tree reasoning to generate effective feature rules without manual search spaces, and it demonstrates consistent performance improvements over competing methods across diverse benchmarks.
In tabular prediction tasks, tree-based models combined with automated feature engineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on a pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments. To address these limitations, we propose a novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature generator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules without manually specifying the search space and provide language-based reasoning information highlighting past experiments as feedback for iterative rule improvements. We use decision trees to convey this reasoning information, as they can be easily represented in natural language, effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) to the LLMs. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outperforming competing automated feature engineering methods. Code is available at https://github.com/jaehyun513/OCTree.