Criticality & Deep Learning I: Generally Weighted Nets
This foundational work aims to understand how critical phenomena might enhance learning in biological and artificial networks, but it is incremental as it represents a first step in a series of investigations.
The paper investigates the role of criticality and universality from phase transitions in deep learning, analyzing theoretical calculations for feed-forward networks and experimental traces in neural networks as an initial step to explore this relationship.
Motivated by the idea that criticality and universality of phase transitions might play a crucial role in achieving and sustaining learning and intelligent behaviour in biological and artificial networks, we analyse a theoretical and a pragmatic experimental set up for critical phenomena in deep learning. On the theoretical side, we use results from statistical physics to carry out critical point calculations in feed-forward/fully connected networks, while on the experimental side we set out to find traces of criticality in deep neural networks. This is our first step in a series of upcoming investigations to map out the relationship between criticality and learning in deep networks.