A reproducible comparative study of categorical kernels for Gaussian process regression, with new clustering-based nested kernels
This work addresses the reproducibility and evaluation challenges in selecting categorical kernels for Gaussian process regression, which is important for researchers and practitioners in machine learning dealing with mixed continuous-categorical data.
The authors tackled the challenge of identifying preferred categorical kernels for Gaussian process regression by conducting a reproducible comparative study of existing methods and proposing new clustering-based nested kernels. Their results showed that nested kernels clearly outperform competitors when datasets exhibit group structure, and their new clustering strategy outperforms other approaches across a large panel of datasets while maintaining low computational cost.
Designing categorical kernels is a major challenge for Gaussian process regression with continuous and categorical inputs. Despite previous studies, it is difficult to identify a preferred method, either because the evaluation metrics, the optimization procedure, or the datasets change depending on the study. In particular, reproducible code is rarely available. The aim of this paper is to provide a reproducible comparative study of all existing categorical kernels on many of the test cases investigated so far. We also propose new evaluation metrics inspired by the optimization community, which provide quantitative rankings of the methods across several tasks. From our results on datasets which exhibit a group structure on the levels of categorical inputs, it appears that nested kernels methods clearly outperform all competitors. When the group structure is unknown or when there is no prior knowledge of such a structure, we propose a new clustering-based strategy using target encodings of categorical variables. We show that on a large panel of datasets, which do not necessarily have a known group structure, this estimation strategy still outperforms other approaches while maintaining low computational cost.