Transformationally Identical and Invariant Convolutional Neural Networks by Combining Symmetric Operations or Input Vectors
This work provides a theoretical framework for building transformationally invariant CNNs, which could enhance robustness in applications like image recognition, but it appears incremental as it builds on existing invariance concepts.
The study tackled the problem of constructing transformationally identical convolutional neural networks (CNNs) by demonstrating that combining symmetric operations or averaging transformed input vectors leads to mathematically equivalent systems under specific conditions, applicable to most CNN architectures.
Transformationally invariant processors constructed by transformed input vectors or operators have been suggested and applied to many applications. In this study, transformationally identical processing based on combining results of all sub-processes with corresponding transformations at one of the processing steps or at the beginning step were found to be equivalent for a given condition. This property can be applied to most convolutional neural network (CNN) systems. Specifically, a transformationally identical CNN can be constructed by arranging internally symmetric operations in parallel with the same transformation family that includes a flatten layer with weights sharing among their corresponding transformation elements. Other transformationally identical CNNs can be constructed by averaging transformed input vectors of the family at the input layer followed by an ordinary CNN process or by a set of symmetric operations. Interestingly, we found that both types of transformationally identical CNN systems are mathematically equivalent by either applying an averaging operation to corresponding elements of all sub-channels before the activation function or without using a non-linear activation function.