Neural networks with superexpressive activations and integer weights
This work addresses the challenge of efficient function approximation in machine learning with constrained network architectures, though it appears incremental as it builds on existing theory with new activation functions.
The paper tackles the problem of approximating continuous functions using neural networks with integer weights and a specific activation function, achieving a convergence rate of order n^(-2β/(2β+d)) log₂ n for Hölder continuous functions with n samples.
An example of an activation function $σ$ is given such that networks with activations $\{σ, \lfloor\cdot\rfloor\}$, integer weights and a fixed architecture depending on $d$ approximate continuous functions on $[0,1]^d$. The range of integer weights required for $\varepsilon$-approximation of Hölder continuous functions is derived, which leads to a convergence rate of order $n^{\frac{-2β}{2β+d}}\log_2n$ for neural network regression estimation of unknown $β$-Hölder continuous function with given $n$ samples.