Universal Approximation Theorem for Input-Connected Multilayer Perceptrons
This work addresses a foundational problem in machine learning theory for researchers and practitioners by providing a theoretical guarantee for a novel neural network architecture, though it appears incremental as it builds on existing universal approximation theorems.
The paper tackles the problem of approximating continuous functions with neural networks by introducing the Input-Connected Multilayer Perceptron (IC-MLP), which includes direct affine connections from raw inputs to hidden neurons, and proves a universal approximation theorem showing that deep IC-MLPs can approximate any continuous function on compact subsets of real spaces if the activation function is nonlinear.
We introduce the Input-Connected Multilayer Perceptron (IC-MLP), a feedforward neural network architecture in which each hidden neuron receives, in addition to the outputs of the preceding layer, a direct affine connection from the raw input. We first study this architecture in the univariate setting and give an explicit and systematic description of IC-MLPs with an arbitrary finite number of hidden layers, including iterated formulas for the network functions. In this setting, we prove a universal approximation theorem showing that deep IC-MLPs can approximate any continuous function on a closed interval of the real line if and only if the activation function is nonlinear. We then extend the analysis to vector-valued inputs and establish a corresponding universal approximation theorem for continuous functions on compact subsets of $\mathbb{R}^n$.