Power Law in Sparsified Deep Neural Networks
This work addresses the understanding of structural similarities between artificial and biological neural networks, which could inform more efficient and brain-inspired AI designs, though it is incremental in linking known concepts to deep learning.
The paper investigated whether sparsified deep neural networks exhibit power law degree distributions, similar to biological neural networks, and found that both multilayer perceptrons and convolutional neural networks show this property. It also proposed an internal preferential attachment model to explain network topology evolution in continual learning, with experiments confirming that new connections follow this process.
The power law has been observed in the degree distributions of many biological neural networks. Sparse deep neural networks, which learn an economical representation from the data, resemble biological neural networks in many ways. In this paper, we study if these artificial networks also exhibit properties of the power law. Experimental results on two popular deep learning models, namely, multilayer perceptrons and convolutional neural networks, are affirmative. The power law is also naturally related to preferential attachment. To study the dynamical properties of deep networks in continual learning, we propose an internal preferential attachment model to explain how the network topology evolves. Experimental results show that with the arrival of a new task, the new connections made follow this preferential attachment process.