Gradient Descent Efficiency Index
This work addresses optimization efficiency for machine learning practitioners in resource-constrained environments where training time directly impacts costs, though it appears incremental as it builds on existing gradient descent methods.
This study tackled the problem of inefficient final iterations in gradient descent by introducing a new efficiency metric Ek that quantifies the effectiveness of each iteration based on relative error change and loss stability. Experimental validation across multiple datasets and models showed that Ek provides valuable insights into convergence behavior and complements traditional performance metrics.
Gradient descent is a widely used iterative algorithm for finding local minima in multivariate functions. However, the final iterations often either overshoot the minima or make minimal progress, making it challenging to determine an optimal stopping point. This study introduces a new efficiency metric, Ek, designed to quantify the effectiveness of each iteration. The proposed metric accounts for both the relative change in error and the stability of the loss function across iterations. This measure is particularly valuable in resource-constrained environments, where costs are closely tied to training time. Experimental validation across multiple datasets and models demonstrates that Ek provides valuable insights into the convergence behavior of gradient descent, complementing traditional performance metrics. The index has the potential to guide more informed decisions in the selection and tuning of optimization algorithms in machine learning applications and be used to compare the "effectiveness" of models relative to each other.