Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
This addresses a foundational problem in deep learning theory for researchers, offering a prescriptive approach to improve generalization, though it appears incremental as it builds on existing gradient dynamics concepts.
The paper tackles the open question of why neural networks generalize well despite overfitting capacity by proposing the Coherent Gradients hypothesis, which posits that gradient similarities during training bias updates to benefit many examples simultaneously, and it suggests a modification to gradient descent that can greatly reduce overfitting.
An open question in the Deep Learning community is why neural networks trained with Gradient Descent generalize well on real datasets even though they are capable of fitting random data. We propose an approach to answering this question based on a hypothesis about the dynamics of gradient descent that we call Coherent Gradients: Gradients from similar examples are similar and so the overall gradient is stronger in certain directions where these reinforce each other. Thus changes to the network parameters during training are biased towards those that (locally) simultaneously benefit many examples when such similarity exists. We support this hypothesis with heuristic arguments and perturbative experiments and outline how this can explain several common empirical observations about Deep Learning. Furthermore, our analysis is not just descriptive, but prescriptive. It suggests a natural modification to gradient descent that can greatly reduce overfitting.