Black-Box Optimization in Machine Learning with Trust Region Based Derivative Free Algorithm
This work addresses black-box optimization problems in machine learning, such as AUC maximization, but appears incremental as it applies an existing trust region method to a specific context.
The paper tackled maximizing the Area Under the ROC Curve (AUC) as a nonsmooth, noisy function in black-box optimization, showing that DFO-TR outperformed Bayesian optimization and random search in tasks like AUC maximization and hyperparameter tuning.
In this work, we utilize a Trust Region based Derivative Free Optimization (DFO-TR) method to directly maximize the Area Under Receiver Operating Characteristic Curve (AUC), which is a nonsmooth, noisy function. We show that AUC is a smooth function, in expectation, if the distributions of the positive and negative data points obey a jointly normal distribution. The practical performance of this algorithm is compared to three prominent Bayesian optimization methods and random search. The presented numerical results show that DFO-TR surpasses Bayesian optimization and random search on various black-box optimization problem, such as maximizing AUC and hyperparameter tuning.