In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
arXiv:1412.6614v4720 citations
Originality Synthesis-oriented
AI Analysis
This addresses the fundamental problem of explaining generalization in deep learning for researchers, but it appears incremental as it builds on existing analogies to matrix factorization.
The paper investigates the role of implicit regularization, distinct from network size, in controlling capacity for multilayer feed-forward networks, suggesting it as a key inductive bias to understand deep learning.
We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.