Response to Promises and Pitfalls of Deep Kernel Learning
This addresses a theoretical issue in deep kernel learning for researchers, clarifying hyperparameter optimization to prevent misinterpretations, but it is incremental as it refines an existing analysis.
The authors respond to a prior claim that in deep kernel learning, reparameterizing a kernel's signal variance leads to a complexity penalty dominating hyperparameter selection, potentially causing overcorrelation; they demonstrate that the reparameterization actually introduces an additional data-fit term, maintaining a balance between data fit and complexity in determining hyperparameters.
This note responds to "Promises and Pitfalls of Deep Kernel Learning" (Ober et al., 2021). The marginal likelihood of a Gaussian process can be compartmentalized into a data fit term and a complexity penalty. Ober et al. (2021) shows that if a kernel can be multiplied by a signal variance coefficient, then reparametrizing and substituting in the maximized value of this parameter sets a reparametrized data fit term to a fixed value. They use this finding to argue that the complexity penalty, a log determinant of the kernel matrix, then dominates in determining the other values of kernel hyperparameters, which can lead to data overcorrelation. By contrast, we show that the reparametrization in fact introduces another data-fit term which influences all other kernel hyperparameters. Thus, a balance between data fit and complexity still plays a significant role in determining kernel hyperparameters.