A Comparison of Machine Learning Methods for Data with High-Cardinality Categorical Variables
This addresses challenges in machine learning for tabular data with many categorical levels, but it is incremental as it compares existing methods.
The study compared machine learning methods for handling high-cardinality categorical variables, finding that models with random effects improved prediction accuracy and tree-boosting with random effects outperformed deep neural networks with random effects.
High-cardinality categorical variables are variables for which the number of different levels is large relative to the sample size of a data set, or in other words, there are few data points per level. Machine learning methods can have difficulties with high-cardinality variables. In this article, we empirically compare several versions of two of the most successful machine learning methods, tree-boosting and deep neural networks, and linear mixed effects models using multiple tabular data sets with high-cardinality categorical variables. We find that, first, machine learning models with random effects have higher prediction accuracy than their classical counterparts without random effects, and, second, tree-boosting with random effects outperforms deep neural networks with random effects.