Equivariant neural networks and piecewise linear representation theory
This provides a theoretical tool for interpreting equivariant neural networks, which is incremental as it builds on existing representation theory concepts.
The paper tackles the problem of interpreting equivariant neural networks by decomposing their layers into simple representations, showing that nonlinear activations like ReLU create piecewise linear maps and lead to a filtration generalizing Fourier series.
Equivariant neural networks are neural networks with symmetry. Motivated by the theory of group representations, we decompose the layers of an equivariant neural network into simple representations. The nonlinear activation functions lead to interesting nonlinear equivariant maps between simple representations. For example, the rectified linear unit (ReLU) gives rise to piecewise linear maps. We show that these considerations lead to a filtration of equivariant neural networks, generalizing Fourier series. This observation might provide a useful tool for interpreting equivariant neural networks.