The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
This work addresses the problem of improving optimization and understanding brain processes for researchers in AI and neuroscience, but it is incremental as it builds on existing ideas without presenting new empirical results.
The essay explores the intersection of deep learning and neuroscience by examining how biological neural development could inspire efficient optimization procedures in large language models, and how these models might serve as models for brain maturation and aging.
In this essay, we explore a point of intersection between deep learning and neuroscience, through the lens of large language models, transfer learning and network compression. Just like perceptual and cognitive neurophysiology has inspired effective deep neural network architectures which in turn make a useful model for understanding the brain, here we explore how biological neural development might inspire efficient and robust optimization procedures which in turn serve as a useful model for the maturation and aging of the brain.