Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks
This work addresses the encoding of data symmetries in neural networks for researchers in machine learning theory, offering a novel perspective but being incremental in its theoretical contributions.
The authors tackled the problem of understanding how symmetries and geometry in input data are encoded within neural networks by introducing a systematic method to derive a generalized neural network and its inverse (ridgelet transform) from joint group invariant functions. They provided a new group-theoretic proof of universality for a wide class of networks, connecting geometric deep learning to abstract harmonic analysis.
The symmetry and geometry of input data are considered to be encoded in the internal data representation inside the neural network, but the specific encoding rule has been less investigated. In this study, we present a systematic method to induce a generalized neural network and its right inverse operator, called the ridgelet transform, from a joint group invariant function on the data-parameter domain. Since the ridgelet transform is an inverse, (1) it can describe the arrangement of parameters for the network to represent a target function, which is understood as the encoding rule, and (2) it implies the universality of the network. Based on the group representation theory, we present a new simple proof of the universality by using Schur's lemma in a unified manner covering a wide class of networks, for example, the original ridgelet transform, formal deep networks, and the dual voice transform. Since traditional universality theorems were demonstrated based on functional analysis, this study sheds light on the group theoretic aspect of the approximation theory, connecting geometric deep learning to abstract harmonic analysis.