Comparative Study on the Performance of Categorical Variable Encoders in Classification and Regression Tasks
This work provides practical guidance for data scientists in fields like fraud detection and disease diagnosis on choosing encoders, but it is incremental as it builds on existing encoder methods without introducing new ones.
This study tackled the problem of selecting categorical variable encoders for machine learning tasks by theoretically proving that one-hot encoders are best for affine transformation models and target encoders for tree-based models, and validated this through experiments on 28 datasets with 14 encoders and 8 models, showing computational results that align with the analysis.
Categorical variables often appear in datasets for classification and regression tasks, and they need to be encoded into numerical values before training. Since many encoders have been developed and can significantly impact performance, choosing the appropriate encoder for a task becomes a time-consuming yet important practical issue. This study broadly classifies machine learning models into three categories: 1) ATI models that implicitly perform affine transformations on inputs, such as multi-layer perceptron neural network; 2) Tree-based models that are based on decision trees, such as random forest; and 3) the rest, such as kNN. Theoretically, we prove that the one-hot encoder is the best choice for ATI models in the sense that it can mimic any other encoders by learning suitable weights from the data. We also explain why the target encoder and its variants are the most suitable encoders for tree-based models. This study conducted comprehensive computational experiments to evaluate 14 encoders, including one-hot and target encoders, along with eight common machine-learning models on 28 datasets. The computational results agree with our theoretical analysis. The findings in this study shed light on how to select the suitable encoder for data scientists in fields such as fraud detection, disease diagnosis, etc.