Statistical Test for Auto Feature Engineering by Selective Inference
This addresses the need for theoretically sound evaluation of AFE features in machine learning pipelines, though it is incremental as it focuses on a specific class of algorithms and linear models.
The paper tackles the problem of assessing the reliability of features generated by Auto Feature Engineering (AFE) algorithms, which often lack theoretical guarantees due to heuristic methods, by proposing a statistical test based on selective inference that provides p-values to control false discovery risks.
Auto Feature Engineering (AFE) plays a crucial role in developing practical machine learning pipelines by automating the transformation of raw data into meaningful features that enhance model performance. By generating features in a data-driven manner, AFE enables the discovery of important features that may not be apparent through human experience or intuition. On the other hand, since AFE generates features based on data, there is a risk that these features may be overly adapted to the data, making it essential to assess their reliability appropriately. Unfortunately, because most AFE problems are formulated as combinatorial search problems and solved by heuristic algorithms, it has been challenging to theoretically quantify the reliability of generated features. To address this issue, we propose a new statistical test for generated features by AFE algorithms based on a framework called selective inference. As a proof of concept, we consider a simple class of tree search-based heuristic AFE algorithms, and consider the problem of testing the generated features when they are used in a linear model. The proposed test can quantify the statistical significance of the generated features in the form of $p$-values, enabling theoretically guaranteed control of the risk of false findings.