SPI-Optimizer: an integral-Separated PI Controller for Stochastic Optimization
This addresses optimization instability in deep learning, offering a practical improvement for training neural networks, though it is incremental as it builds on existing PI controller methods.
The paper tackles the oscillation problem in momentum-based optimizers by proposing SPI-Optimizer, an integral-separated PI controller that eliminates oscillation without extra hyperparameters, achieving up to 40% faster convergence and 27.5% error reduction on datasets like CIFAR10.
To overcome the oscillation problem in the classical momentum-based optimizer, recent work associates it with the proportional-integral (PI) controller, and artificially adds D term producing a PID controller. It suppresses oscillation with the sacrifice of introducing extra hyper-parameter. In this paper, we start by analyzing: why momentum-based method oscillates about the optimal point? and answering that: the fluctuation problem relates to the lag effect of integral (I) term. Inspired by the conditional integration idea in classical control society, we propose SPI-Optimizer, an integral-Separated PI controller based optimizer WITHOUT introducing extra hyperparameter. It separates momentum term adaptively when the inconsistency of current and historical gradient direction occurs. Extensive experiments demonstrate that SPIOptimizer generalizes well on popular network architectures to eliminate the oscillation, and owns competitive performance with faster convergence speed (up to 40% epochs reduction ratio ) and more accurate classification result on MNIST, CIFAR10, and CIFAR100 (up to 27.5% error reduction ratio) than the state-of-the-art methods.