Failures of Gradient-Based Deep Learning
This work highlights critical failures in widely used deep learning methods, which is important for researchers and practitioners to understand and address.
The paper identifies four simple problems where gradient-based deep learning algorithms fail or struggle, providing experimental evidence and theoretical explanations for these limitations.
In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art. However, it is important, for both theoreticians and practitioners, to gain a deeper understanding of the difficulties and limitations associated with common approaches and algorithms. We describe four types of simple problems, for which the gradient-based algorithms commonly used in deep learning either fail or suffer from significant difficulties. We illustrate the failures through practical experiments, and provide theoretical insights explaining their source, and how they might be remedied.