Neurons learn slower than they think
This work addresses optimization challenges in machine learning for researchers, but it appears incremental as it builds on known dynamics without broad SOTA impact.
The study tackled the problem of balancing convergence rate and generalization error in gradient-based optimization, finding that maximizing test accuracy often requires a larger learning rate than minimizing training loss, and introduced differential capability to analyze this dynamic, showing that higher convergence rates slow capability growth while lower rates accelerate it and decay.
Recent studies revealed complex convergence dynamics in gradient-based methods, which has been little understood so far. Changing the step size to balance between high convergence rate and small generalization error may not be sufficient: maximizing the test accuracy usually requires a larger learning rate than minimizing the training loss. To explore the dynamic bounds of convergence rate, this study introduces \textit{differential capability} into an optimization process, which measures whether the test accuracy increases as fast as a model approaches the decision boundary in a classification problem. The convergence analysis showed that: 1) a higher convergence rate leads to slower capability growth; 2) a lower convergence rate results in faster capability growth and decay; 3) regulating a convergence rate in either direction reduces differential capability.