Distributional encoding for Gaussian process regression with qualitative inputs
This addresses a bottleneck in Gaussian Process regression and Bayesian optimization for applications with categorical inputs, offering an incremental improvement over existing encoding methods.
The paper tackles the challenge of building predictive and computationally efficient Gaussian Process regression when input variables are categorical, by proposing a distributional encoding method that uses all target variable samples per category. The approach demonstrates state-of-the-art predictive performance on synthetic and real-world datasets.
Gaussian Process (GP) regression is a popular and sample-efficient approach for many engineering applications, where observations are expensive to acquire, and is also a central ingredient of Bayesian optimization (BO), a highly prevailing method for the optimization of black-box functions. However, when all or some input variables are categorical, building a predictive and computationally efficient GP remains challenging. Starting from the naive target encoding idea, where the original categorical values are replaced with the mean of the target variable for that category, we propose a generalization based on distributional encoding (DE) which makes use of all samples of the target variable for a category. To handle this type of encoding inside the GP, we build upon recent results on characteristic kernels for probability distributions, based on the maximum mean discrepancy and the Wasserstein distance. We also discuss several extensions for classification, multi-task learning and incorporation or auxiliary information. Our approach is validated empirically, and we demonstrate state-of-the-art predictive performance on a variety of synthetic and real-world datasets. DE is naturally complementary to recent advances in BO over discrete and mixed-spaces.