Bayesian Optimization under Uncertainty for Training a Scale Parameter in Stochastic Models
This work addresses the problem of efficient hyperparameter tuning in stochastic models for computational engineering, representing an incremental improvement with specific computational gains.
The paper tackles hyperparameter tuning under uncertainty by introducing a Bayesian optimization framework that uses a statistical surrogate and closed-form expressions to reduce computational cost, achieving up to a 40-fold reduction in data points and computational cost compared to conventional Monte Carlo methods.
Hyperparameter tuning is a challenging problem especially when the system itself involves uncertainty. Due to noisy function evaluations, optimization under uncertainty can be computationally expensive. In this paper, we present a novel Bayesian optimization framework tailored for hyperparameter tuning under uncertainty, with a focus on optimizing a scale- or precision-type parameter in stochastic models. The proposed method employs a statistical surrogate for the underlying random variable, enabling analytical evaluation of the expectation operator. Moreover, we derive a closed-form expression for the optimizer of the random acquisition function, which significantly reduces computational cost per iteration. Compared with a conventional one-dimensional Monte Carlo-based optimization scheme, the proposed approach requires 40 times fewer data points, resulting in up to a 40-fold reduction in computational cost. We demonstrate the effectiveness of the proposed method through two numerical examples in computational engineering.