Towards Understanding Generalization via Decomposing Excess Risk Dynamics
This work addresses the fundamental issue of generalization in machine learning, particularly for deep learning, by offering a more precise theoretical analysis, though it is incremental in refining existing stability-based approaches.
The paper tackles the problem of explaining generalization in overparameterized models by proposing a decomposition framework that improves stability-based bounds through fine-grained analysis of signal and noise components, achieving better performance in linear and non-linear regimes.
Generalization is one of the fundamental issues in machine learning. However, traditional techniques like uniform convergence may be unable to explain generalization under overparameterization. As alternative approaches, techniques based on stability analyze the training dynamics and derive algorithm-dependent generalization bounds. Unfortunately, the stability-based bounds are still far from explaining the surprising generalization in deep learning since neural networks usually suffer from unsatisfactory stability. This paper proposes a novel decomposition framework to improve the stability-based bounds via a more fine-grained analysis of the signal and noise, inspired by the observation that neural networks converge relatively slowly when fitting noise (which indicates better stability). Concretely, we decompose the excess risk dynamics and apply the stability-based bound only on the noise component. The decomposition framework performs well in both linear regimes (overparameterized linear regression) and non-linear regimes (diagonal matrix recovery). Experiments on neural networks verify the utility of the decomposition framework.