Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables
This addresses optimization challenges in deep learning for researchers and practitioners, but it appears incremental as it builds on existing methods like auxiliary variables.
The paper tackles the inefficiency of gradient descent in deep learning due to non-convex loss functions and vanishing gradients by introducing auxiliary variables and self-adaptive weights to reformulate the loss for easier optimization, with numerical experiments showing effectiveness and robustness.
In this paper, we develop a new optimization framework for the least squares learning problem via fully connected neural networks or physics-informed neural networks. The gradient descent sometimes behaves inefficiently in deep learning because of the high non-convexity of loss functions and the vanishing gradient issue. Our idea is to introduce auxiliary variables to separate the layers of the deep neural networks and reformulate the loss functions for ease of optimization. We design the self-adaptive weights to preserve the consistency between the reformulated loss and the original mean squared loss, which guarantees that optimizing the new loss helps optimize the original problem. Numerical experiments are presented to verify the consistency and show the effectiveness and robustness of our models over gradient descent.