Synthesizing Deep Neural Network Architectures using Biological Synaptic Strength Distributions
This addresses the problem of data-hungry deep learning for researchers and practitioners working with limited datasets, though it appears incremental as an exploratory study.
The researchers investigated whether deep neural networks with convolutional layer synaptic strengths drawn from biological distributions (like log-normal or correlated center-surround) could perform well on small datasets, finding that such architectures maintained or boosted modeling performance without requiring extensive data or standard training procedures.
In this work, we perform an exploratory study on synthesizing deep neural networks using biological synaptic strength distributions, and the potential influence of different distributions on modelling performance particularly for the scenario associated with small data sets. Surprisingly, a CNN with convolutional layer synaptic strengths drawn from biologically-inspired distributions such as log-normal or correlated center-surround distributions performed relatively well suggesting a possibility for designing deep neural network architectures that do not require many data samples to learn, and can sidestep current training procedures while maintaining or boosting modelling performance.