Toward a Physics of Deep Learning and Brains
This provides a theoretical foundation for understanding and potentially improving both biological and artificial neural networks, though it is incremental as it builds on existing physics concepts.
The authors tackled the problem of finding a unified theoretical framework for deep neural networks and brains by showing that equations from non-equilibrium statistical physics, used for neuronal avalanches, apply to deep networks, revealing that networks learn best in a quasi-critical regime with maximal susceptibility predicting performance.
Deep neural networks and brains both learn and share superficial similarities: processing nodes are likened to neurons and adjustable weights are likened to modifiable synapses. But can a unified theoretical framework be found to underlie them both? Here we show that the equations used to describe neuronal avalanches in living brains can also be applied to cascades of activity in deep neural networks. These equations are derived from non-equilibrium statistical physics and show that deep neural networks learn best when poised between absorbing and active phases. Because these networks are strongly driven by inputs, however, they do not operate at a true critical point but within a quasi-critical regime -- one that still approximately satisfies crackling noise scaling relations. By training networks with different initializations, we show that maximal susceptibility is a more reliable predictor of learning than proximity to the critical point itself. This provides a blueprint for engineering improved network performance. Finally, using finite-size scaling we identify distinct universality classes, including Barkhausen noise and directed percolation. This theoretical framework demonstrates that universal features are shared by both biological and artificial neural networks.