Scalable Differentially Private Bayesian Optimization
This work addresses the problem of privately optimizing high-dimensional parameter spaces for users of large machine learning models, which is an incremental yet significant improvement.
The authors tackled the problem of scaling Bayesian Optimization to high-dimensional problems while preserving Differential Privacy, achieving exponential convergence to a locally optimal parameter configuration with a natural privacy error. Their method empirically outperforms existing methods in high-dimensional hyperparameter settings.
In recent years, there has been much work on scaling Bayesian Optimization to high-dimensional problems, for example hyperparameter tuning in large machine learning models. These scalable methods have been successful, finding high objective values much more quickly than traditional global Bayesian Optimization or random search-based methods. At the same time, these large models often use sensitive data, but preservation of Differential Privacy has not scaled alongside these modern Bayesian Optimization procedures. Here we develop a method to privately optimize potentially high-dimensional parameter spaces using privatized Gradient Informative Bayesian Optimization. Our theoretical results show that under suitable conditions, our method converges exponentially fast to a locally optimal parameter configuration, up to a natural privacy error. Moreover, regardless of whether the assumptions are satisfied, we prove that our algorithm maintains privacy and empirically display superior performance to existing methods in the high-dimensional hyperparameter setting.