Gradient Descent Algorithm Survey
It provides a standardized reference for selecting and tuning optimization algorithms in academic and engineering contexts, addressing optimization challenges across different model scales and training scenarios.
This survey analyzes five gradient descent algorithms (SGD, Mini-batch SGD, Momentum, Adam, and Lion) to understand their advantages, limitations, and practical recommendations for optimization in deep learning.
Focusing on the practical configuration needs of optimization algorithms in deep learning, this article concentrates on five major algorithms: SGD, Mini-batch SGD, Momentum, Adam, and Lion. It systematically analyzes the core advantages, limitations, and key practical recommendations of each algorithm. The research aims to gain an in-depth understanding of these algorithms and provide a standardized reference for the reasonable selection, parameter tuning, and performance improvement of optimization algorithms in both academic research and engineering practice, helping to solve optimization challenges in different scales of models and various training scenarios.