Differentiable programming: Generalization, characterization and limitations of deep learning
This work provides a theoretical framework for understanding deep learning's scope and limitations, which is foundational for AI researchers but incremental in its contributions.
The paper defines differentiable programming as a generalization of deep learning and analyzes program characteristics that incorporate problem structure, evaluating them on a graph dataset. It also discusses inherent limitations of deep learning and differentiable programs, identifying key challenges for advancing AI.
In the past years, deep learning models have been successfully applied in several cognitive tasks. Originally inspired by neuroscience, these models are specific examples of differentiable programs. In this paper we define and motivate differentiable programming, as well as specify some program characteristics that allow us to incorporate the structure of the problem in a differentiable program. We analyze different types of differentiable programs, from more general to more specific, and evaluate, for a specific problem with a graph dataset, its structure and knowledge with several differentiable programs using those characteristics. Finally, we discuss some inherent limitations of deep learning and differentiable programs, which are key challenges in advancing artificial intelligence, and then analyze possible solutions