Recurrent neural networks approximate continuous functions
For the theory of neural network approximation, this work shows that fixed networks can achieve arbitrary accuracy by increasing runtime, shifting the resource from network size to time.
The paper proves that any continuous function on [-1,1] can be uniformly approximated by the time evolution of a single fixed-weight ReLU recurrent neural network, with accuracy improving as the network runs longer. It establishes convergence rates matching polynomial approximation and provides minimax lower bounds showing runtime is unavoidable.
Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuous function on [-1,1]. More precisely, each such function is uniformly approximated by the time evolution of a single ReLU recurrent neural network with fixed weights and fixed hidden dimension. The mechanism behind the construction is a new intermediate model, the Turing machine with neural units (TMNU). This model retains the algorithmic freedom needed to implement polynomial approximation schemes, while remaining rigid enough to be simulated by RNNs with explicit bounds on hidden dimension and weight magnitude. The resulting convergence rates reflect the underlying polynomial approximation rates. We complement the construction with minimax lower bounds showing that runtime is not merely a proof artifact, but an unavoidable resource in this fixed-network approximation paradigm.