Explore the Loss space with Hill-ADAM
This addresses the challenge of finding global minima in optimization for machine learning, though it appears incremental as it builds on ADAM with a deterministic exploration method.
The paper tackles the problem of escaping local minima in optimization by introducing Hill-ADAM, an optimizer that alternates between error minimization and maximization to explore loss spaces, resulting in improved performance tested on 5 loss functions and 12 image color correction instances.
This paper introduces Hill-ADAM. Hill-ADAM is an optimizer with its focus towards escaping local minima in prescribed loss landscapes to find the global minimum. Hill-ADAM escapes minima by deterministically exploring the state space. This eliminates uncertainty from random gradient updates in stochastic algorithms while seldom converging at the first minimum that visits. In the paper we first derive an analytical approximation of the ADAM Optimizer step size at a particular model state. From there define the primary condition determining ADAM limitations in escaping local minima. The proposed optimizer algorithm Hill-ADAM alternates between error minimization and maximization. It maximizes to escape the local minimum and minimizes again afterward. This alternation provides an overall exploration throughout the loss space. This allows the deduction of the global minimum's state. Hill-ADAM was tested with 5 loss functions and 12 amber-saturated to cooler-shade image color correction instances.