Activation Functions for "A Feedforward Unitary Equivariant Neural Network"
This work addresses the limitation of activation functions in unitary equivariant neural networks, offering a more flexible design for researchers in equivariant deep learning, though it is incremental.
The authors generalized three previously proposed activation functions for a feedforward unitary equivariant neural network into a single functional form, which maintains unitary equivariance and provides greater flexibility in network design.
In our previous work [Ma and Chan (2023)], we presented a feedforward unitary equivariant neural network. We proposed three distinct activation functions tailored for this network: a softsign function with a small residue, an identity function, and a Leaky ReLU function. While these functions demonstrated the desired equivariance properties, they limited the neural network's architecture. This short paper generalises these activation functions to a single functional form. This functional form represents a broad class of functions, maintains unitary equivariance, and offers greater flexibility for the design of equivariant neural networks.