Hyper-parameter estimation method with particle swarm optimization
This work provides an incremental improvement for machine learning practitioners seeking more efficient hyper-parameter optimization.
This paper proposes a method that integrates Particle Swarm Optimization (PSO) within the Bayesian Optimization (BO) framework to estimate hyper-parameters. By using PSO to optimize the acquisition function in BO, the method demonstrates improved performance on several benchmark problems in both classification and regression models.
Particle swarm optimization (PSO) method cannot be directly used in the problem of hyper-parameter estimation since the mathematical formulation of the mapping from hyper-parameters to loss function or generalization accuracy is unclear. Bayesian optimization (BO) framework is capable of converting the optimization of the hyper-parameters into the optimization of an acquisition function. The acquisition function is non-convex and multi-peak. So the problem can be better solved by the PSO. The proposed method in this paper uses the particle swarm method to optimize the acquisition function in the BO framework to get better hyper-parameters. The performances of proposed method in both of the classification and regression models are evaluated and demonstrated. The results on several benchmark problems are improved.