Pitfalls and Remedies for Multi-Task Bayesian Optimization
For practitioners of Bayesian optimization using transfer learning, the paper reveals fundamental limitations of the standard multi-task GP and provides conservative fixes, though the problem persists in complex scenarios.
The paper identifies that multi-task Gaussian processes misestimate cross-task correlations even in simple affinely related tasks, and proposes three remedies (modeling per-task parameters, non-negative correlations, co-located designs) that recover baseline performance on simple instances but fail on harder ones.
Bayesian optimization routinely warm-starts a target experiment with data from related source tasks, and the multi-task Gaussian process is the textbook surrogate for the job. We revisit this default in a controlled setting and find that it misestimates the cross-task correlation even in the simplest non-trivial case, affinely related source and target tasks, where a working transfer learning method should obviously succeed. We trace the failure to two independent structural mechanisms. Per-task standardization, the textbook fix for the affine slice ambiguity, propagates a finite-sample alignment error into the recovered correlation. The marginal likelihood itself identifies the correlation only at a per-sample rate that a Gaussian process at non-overlapping designs further dilutes. We propose three conservative remedies that follow from the analysis: promoting per-task means and scales to model parameters, restricting the task covariance to non-negative correlations, and co-locating part of the source and target designs. Across synthetic multi-task problems and surrogate-based hyperparameter tuning transfer, these remedies recover the target-only baseline on the simple instances, while the broader failure persists on harder instances and across most rank-based and latent-context variants.