Omnipresent Yet Overlooked: Heat Kernels in Combinatorial Bayesian Optimization
This work addresses the need for effective kernels in combinatorial Bayesian Optimization, which is incremental as it unifies existing methods rather than introducing a new paradigm.
The paper tackled the problem of unifying combinatorial kernels in Bayesian Optimization by developing a framework based on heat kernels, showing that many existing kernels are related or equivalent to them, and demonstrated that a simple pipeline using heat kernels achieves state-of-the-art results, matching or outperforming complex algorithms.
Bayesian Optimization (BO) has the potential to solve various combinatorial tasks, ranging from materials science to neural architecture search. However, BO requires specialized kernels to effectively model combinatorial domains. Recent efforts have introduced several combinatorial kernels, but the relationships among them are not well understood. To bridge this gap, we develop a unifying framework based on heat kernels, which we derive in a systematic way and express as simple closed-form expressions. Using this framework, we prove that many successful combinatorial kernels are either related or equivalent to heat kernels, and validate this theoretical claim in our experiments. Moreover, our analysis confirms and extends the results presented in Bounce: certain algorithms' performance decreases substantially when the unknown optima of the function do not have a certain structure. In contrast, heat kernels are not sensitive to the location of the optima. Lastly, we show that a fast and simple pipeline, relying on heat kernels, is able to achieve state-of-the-art results, matching or even outperforming certain slow or complex algorithms.