DiffPrep: Differentiable Data Preprocessing Pipeline Search for Learning over Tabular Data
This addresses the time-consuming and expert-dependent nature of data preprocessing in AutoML for tabular data, offering an efficient solution with incremental improvements over existing methods.
The paper tackles the problem of automating data preprocessing for tabular data by proposing DiffPrep, a method that searches for optimal preprocessing pipelines using gradient descent, achieving the best test accuracy on 15 out of 18 datasets and improving accuracy by up to 6.6 percentage points.
Data preprocessing is a crucial step in the machine learning process that transforms raw data into a more usable format for downstream ML models. However, it can be costly and time-consuming, often requiring the expertise of domain experts. Existing automated machine learning (AutoML) frameworks claim to automate data preprocessing. However, they often use a restricted search space of data preprocessing pipelines which limits the potential performance gains, and they are often too slow as they require training the ML model multiple times. In this paper, we propose DiffPrep, a method that can automatically and efficiently search for a data preprocessing pipeline for a given tabular dataset and a differentiable ML model such that the performance of the ML model is maximized. We formalize the problem of data preprocessing pipeline search as a bi-level optimization problem. To solve this problem efficiently, we transform and relax the discrete, non-differential search space into a continuous and differentiable one, which allows us to perform the pipeline search using gradient descent with training the ML model only once. Our experiments show that DiffPrep achieves the best test accuracy on 15 out of the 18 real-world datasets evaluated and improves the model's test accuracy by up to 6.6 percentage points.