Deep Neural Network Approximation of Invariant Functions through Dynamical Systems
This work addresses the need for approximation guarantees in neural network architectures for symmetric functions, which is incremental as it extends existing theory to broader symmetry requirements.
The paper tackles the problem of approximating invariant functions under permutations using flow maps of dynamical systems, proving sufficient conditions for universal approximation that apply to deep residual networks with symmetry constraints and guide architecture design for new symmetries.
We study the approximation of functions which are invariant with respect to certain permutations of the input indices using flow maps of dynamical systems. Such invariant functions includes the much studied translation-invariant ones involving image tasks, but also encompasses many permutation-invariant functions that finds emerging applications in science and engineering. We prove sufficient conditions for universal approximation of these functions by a controlled equivariant dynamical system, which can be viewed as a general abstraction of deep residual networks with symmetry constraints. These results not only imply the universal approximation for a variety of commonly employed neural network architectures for symmetric function approximation, but also guide the design of architectures with approximation guarantees for applications involving new symmetry requirements.