Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoders
This addresses the challenge of stable and expressive learning in spiking neural networks for machine learning applications, though it appears incremental by building on existing spiking network frameworks.
The paper tackles the learning problem in spiking neural networks by introducing affine encoders and decoders with positive synaptic weights, achieving continuous parameter dependence for stable training and competitive results on benchmarks, including rate-optimal approximation of smooth functions and dimension-independent approximation of Barron regular functions.
We study the learning problem associated with spiking neural networks. Specifically, we focus on spiking neural networks composed of simple spiking neurons having only positive synaptic weights, equipped with an affine encoder and decoder; we refer to these as affine spiking neural networks. These neural networks are shown to depend continuously on their parameters, which facilitates classical covering number-based generalization statements and supports stable gradient-based training. We demonstrate that the positivity of the weights enables a wide range of expressivity results, including rate-optimal approximation of smooth functions and dimension-independent approximation of Barron regular functions. In particular, we show in theory and simulations that affine spiking neural networks are capable of approximating shallow ReLU neural networks. Furthermore, we apply these affine spiking neural networks to standard machine learning benchmarks and reach competitive results. Finally, we observe that from a generalization perspective, contrary to feedforward neural networks or previous results for general spiking neural networks, the depth has little to no adverse effect on the generalization capabilities.