Training Neural Networks by Optimizing Neuron Positions
This addresses efficiency challenges for deployment on edge devices or real-time systems, offering a biologically inspired approach with potential for visualization, though it appears incremental as it builds on existing neural network paradigms.
The paper tackles the problem of high computational complexity and parameter counts in deep neural networks for resource-constrained environments by proposing a parameter-efficient architecture where neurons are embedded in Euclidean space, with synaptic weights determined by spatial distance, achieving competitive performance on MNIST and maintaining performance at over 80% sparsity.
The high computational complexity and increasing parameter counts of deep neural networks pose significant challenges for deployment in resource-constrained environments, such as edge devices or real-time systems. To address this, we propose a parameter-efficient neural architecture where neurons are embedded in Euclidean space. During training, their positions are optimized and synaptic weights are determined as the inverse of the spatial distance between connected neurons. These distance-dependent wiring rules replace traditional learnable weight matrices and significantly reduce the number of parameters while introducing a biologically inspired inductive bias: connection strength decreases with spatial distance, reflecting the brain's embedding in three-dimensional space where connections tend to minimize wiring length. We validate this approach for both multi-layer perceptrons and spiking neural networks. Through a series of experiments, we demonstrate that these spatially embedded neural networks achieve a performance competitive with conventional architectures on the MNIST dataset. Additionally, the models maintain performance even at pruning rates exceeding 80% sparsity, outperforming traditional networks with the same number of parameters under similar conditions. Finally, the spatial embedding framework offers an intuitive visualization of the network structure.