AYLA: Amplifying Gradient Sensitivity via Loss Transformation in Non-Convex Optimization
This provides a model-agnostic enhancement to optimization methods for deep neural network training, though it appears incremental as it builds on existing SGD and ADAM frameworks.
The paper tackled the problem of balancing adaptability and efficiency in high-dimensional, non-convex optimization for deep learning by introducing AYLA, a framework that transforms the loss function to amplify gradient sensitivity, resulting in faster convergence and improved stability compared to SGD and ADAM on tasks like MNIST and CIFAR-100.
Stochastic Gradient Descent (SGD) and its variants, such as ADAM, are foundational to deep learning optimization, adjusting model parameters through fixed or adaptive learning rates based on loss function gradients. However, these methods often struggle to balance adaptability and efficiency in high-dimensional, non-convex settings. This paper introduces AYLA, a novel optimization framework that enhances training dynamics via loss function transformation. AYLA applies a tunable power-law transformation to the loss, preserving critical points while scaling loss values to amplify gradient sensitivity and accelerate convergence. Additionally, we propose an effective learning rate that dynamically adapts to the transformed loss, further improving optimization efficiency. Empirical evaluations on minimizing a synthetic non-convex polynomial, solving a non-convex curve-fitting task, and performing digit classification (MNIST) and image recognition (CIFAR-100) demonstrate that AYLA consistently outperforms SGD and ADAM in both convergence speed and training stability. By reshaping the loss landscape, AYLA provides a model-agnostic enhancement to existing optimization methods, offering a promising advancement in deep neural network training.