Quiver neural networks
This work provides a theoretical framework for analyzing neural network architectures, which could benefit researchers in machine learning, but it appears incremental as it builds on existing mathematical concepts without demonstrating broad practical impact.
The authors tackled the analysis of neural network connectivity architectures by introducing quiver neural networks, a theoretical approach inspired by quiver representation theory, and proved a lossless model compression algorithm for networks with rescaling activations, showing equivalence in training between compressed and original models under specific conditions.
We develop a uniform theoretical approach towards the analysis of various neural network connectivity architectures by introducing the notion of a quiver neural network. Inspired by quiver representation theory in mathematics, this approach gives a compact way to capture elaborate data flows in complex network architectures. As an application, we use parameter space symmetries to prove a lossless model compression algorithm for quiver neural networks with certain non-pointwise activations known as rescaling activations. In the case of radial rescaling activations, we prove that training the compressed model with gradient descent is equivalent to training the original model with projected gradient descent.