A Particle-based Sparse Gaussian Process Optimizer
This addresses inefficiencies in optimization for neural networks, particularly in non-convex and high-dimensional tasks like image classification, though it appears incremental as it builds on existing particle-swarm methods.
The paper tackles the problem of neural network optimization getting stuck in local minima by introducing a particle-swarm-based framework that uses Gaussian Process Regression to learn the descent dynamics, resulting in improved ability to escape local minima compared to state-of-the-art optimizers in non-convex problems.
Task learning in neural networks typically requires finding a globally optimal minimizer to a loss function objective. Conventional designs of swarm based optimization methods apply a fixed update rule, with possibly an adaptive step-size for gradient descent based optimization. While these methods gain huge success in solving different optimization problems, there are some cases where these schemes are either inefficient or suffering from local-minimum. We present a new particle-swarm-based framework utilizing Gaussian Process Regression to learn the underlying dynamical process of descent. The biggest advantage of this approach is greater exploration around the current state before deciding a descent direction. Empirical results show our approach can escape from the local minima compare with the widely-used state-of-the-art optimizers when solving non-convex optimization problems. We also test our approach under high-dimensional parameter space case, namely, image classification task.