ConformalLayers: A non-linear sequential neural network with associative layers
This addresses efficiency issues for deep learning practitioners by potentially reducing inference time and resource usage in CNNs, though it appears incremental as it builds on existing CNN architectures.
The authors tackled the problem of high computational cost and memory footprint in deep convolutional neural networks by introducing a new non-linear activation function that enables associativity between sequential layers, allowing all layers to be combined into a single operation and making inference cost constant regardless of network depth.
Convolutional Neural Networks (CNNs) have been widely applied. But as the CNNs grow, the number of arithmetic operations and memory footprint also increase. Furthermore, typical non-linear activation functions do not allow associativity of the operations encoded by consecutive layers, preventing the simplification of intermediate steps by combining them. We present a new activation function that allows associativity between sequential layers of CNNs. Even though our activation function is non-linear, it can be represented by a sequence of linear operations in the conformal model for Euclidean geometry. In this domain, operations like, but not limited to, convolution, average pooling, and dropout remain linear. We take advantage of associativity to combine all the "conformal layers" and make the cost of inference constant regardless of the depth of the network.