A General Method for Proving Networks Universal Approximation Property
This provides a foundational theoretical tool for researchers in machine learning to streamline proofs and unify understanding across network families, though it is incremental as it builds on existing approximation theory.
The paper tackles the problem of proving universal approximation properties for diverse deep learning architectures by introducing a general and modular framework based on Universal Approximation Modules (UAMs), showing that any network composed of such modules inherently retains universal approximation and enabling a unified analysis across architectures.
Deep learning architectures are highly diverse. To prove their universal approximation properties, existing works typically rely on model-specific proofs. Generally, they construct a dedicated mathematical formulation for each architecture (e.g., fully connected networks, CNNs, or Transformers) and then prove their universal approximability. However, this approach suffers from two major limitations: first, every newly proposed architecture often requires a completely new proof from scratch; second, these proofs are largely isolated from one another, lacking a common analytical foundation. This not only incurs significant redundancy but also hinders unified theoretical understanding across different network families. To address these issues, this paper proposes a general and modular framework for proving universal approximation. We define a basic building block (comprising one or multiple layers) that possesses the universal approximation property as a Universal Approximation Module (UAM). Under this condition, we show that any deep network composed of such modules inherently retains the universal approximation property. Moreover, the overall approximation process can be interpreted as a progressive refinement across modules. This perspective not only unifies the analysis of diverse architectures but also enables a step-by-step understanding of how expressive power evolves through the network.