An Open-Source Training Dataset for Foundation Models for Black-box Optimization
This work addresses the lack of large-scale, real-world, public pre-training data for foundation models in black-box optimization, enabling reproducible research and improved generalization.
The authors introduce BBO-Pile, the first open-source dataset of over 500K optimization trajectories across 3095 black-box functions, and train foundation models (2M-80M parameters) that demonstrate large-scale pre-training is effective for imitating black-box optimization methods.
Most black-box optimization methods require extensive hyperparameter tuning, often limiting their ability to generalize across different optimization domains. Foundation models for black-box optimization that learn optimization principles from a large collection of optimization trajectories offer a promising alternative, with the potential to outperform manually designed methods across diverse problem classes. However, prior work has either relied on non-public datasets or on purely synthetic data, limiting reproducibility and generalization to real-world problems. As a result, progress in this area has been constrained by the lack of large-scale, real-world, publicly available pre-training data. We introduce BBO-Pile, the first open-source dataset comprising over 500K optimization trajectories evaluated across 3095 different black-boxes for different optimizers, which represents by far the largest public dataset for this task. Using this dataset, we train a family of foundation models at multiple scales, ranging from 2M to 80M parameters and from 200M to 2B training tokens, and study their scaling behavior with respect to compute. Our results demonstrate that large-scale pre-training is a viable and effective approach to imitate black-box optimization methods, paving the way for future research in this direction.