LocoProp: Enhancing BackProp via Local Loss Optimization
This work addresses the inefficiency of second-order optimizers for deep neural networks in low-budget setups, offering an incremental improvement by bridging the gap with first-order methods.
The paper tackles the problem of second-order optimization methods' high computational and memory costs by introducing LocoProp, a layerwise loss construction framework that uses first-order optimizers to achieve performance closer to second-order methods, as validated on benchmark models and datasets.
Second-order methods have shown state-of-the-art performance for optimizing deep neural networks. Nonetheless, their large memory requirement and high computational complexity, compared to first-order methods, hinder their versatility in a typical low-budget setup. This paper introduces a general framework of layerwise loss construction for multilayer neural networks that achieves a performance closer to second-order methods while utilizing first-order optimizers only. Our methodology lies upon a three-component loss, target, and regularizer combination, for which altering each component results in a new update rule. We provide examples using squared loss and layerwise Bregman divergences induced by the convex integral functions of various transfer functions. Our experiments on benchmark models and datasets validate the efficacy of our new approach, reducing the gap between first-order and second-order optimizers.