Additive function approximation in the brain
This work provides insights into how simple brain architectures can achieve powerful function approximation, potentially aiding computational neuroscientists, but it is incremental in extending kernel theories to sparse networks.
The paper tackles the problem of understanding function approximation in sparsely connected neural networks, such as those in the brain, by showing that networks with d inputs per neuron are equivalent to additive models of order d, which limits the curse of dimensionality and provides stability to noise.
Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons. For a new understanding of such networks, we study the function spaces induced by sparse random features and characterize what functions may and may not be learned. A network with $d$ inputs per neuron is found to be equivalent to an additive model of order $d$, whereas with a degree distribution the network combines additive terms of different orders. We identify three specific advantages of sparsity: additive function approximation is a powerful inductive bias that limits the curse of dimensionality, sparse networks are stable to outlier noise in the inputs, and sparse random features are scalable. Thus, even simple brain architectures can be powerful function approximators. Finally, we hope that this work helps popularize kernel theories of networks among computational neuroscientists.