ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimization
This provides a reusable, general-purpose solution for efficient black-box optimization, addressing the need for robust and transferable methods in fields like engineering and machine learning, though it builds incrementally on existing reinforcement learning and synthetic data techniques.
The paper tackled the problem of sample-efficient global optimization for expensive, derivative-free black-box functions by introducing ZeroShotOpt, a pretrained model that achieves robust zero-shot generalization, matching or surpassing the sample efficiency of leading optimizers like Bayesian optimization across benchmarks from 2D to 20D.
Global optimization of expensive, derivative-free black-box functions requires extreme sample efficiency. While Bayesian optimization (BO) is the current state-of-the-art, its performance hinges on surrogate and acquisition function hyper-parameters that are often hand-tuned and fail to generalize across problem landscapes. We present ZeroShotOpt, a general-purpose, pretrained model for continuous black-box optimization tasks ranging from 2D to 20D. Our approach leverages offline reinforcement learning on large-scale optimization trajectories collected from 12 BO variants. To scale pretraining, we generate millions of synthetic Gaussian process-based functions with diverse landscapes, enabling the model to learn transferable optimization policies. As a result, ZeroShotOpt achieves robust zero-shot generalization on a wide array of unseen benchmarks, matching or surpassing the sample efficiency of leading global optimizers, including BO, while also offering a reusable foundation for future extensions and improvements. Our open-source code, dataset, and model are available at: https://github.com/jamisonmeindl/zeroshotopt