The Role Of Biology In Deep Learning
This work provides incremental insights into biologically-inspired deep learning techniques, primarily relevant for researchers in neural network robustness and continual learning.
This paper investigates whether weight sparsity improves neural network robustness to image noise, finding that sparsity alone does not enhance noise robustness, and extends existing continual learning methods using sparsity and dendrite modeling to more challenging convolutional network tasks.
Artificial neural networks took a lot of inspiration from their biological counterparts in becoming our best machine perceptual systems. This work summarizes some of that history and incorporates modern theoretical neuroscience into experiments with artificial neural networks from the field of deep learning. Specifically, iterative magnitude pruning is used to train sparsely connected networks with 33x fewer weights without loss in performance. These are used to test and ultimately reject the hypothesis that weight sparsity alone improves image noise robustness. Recent work mitigated catastrophic forgetting using weight sparsity, activation sparsity, and active dendrite modeling. This paper replicates those findings, and extends the method to train convolutional neural networks on a more challenging continual learning task. The code has been made publicly available.