Function approximation by deep neural networks with parameters $\{0,\pm \frac{1}{2}, \pm 1, 2\}$
This work addresses the challenge of reducing parameter precision in neural networks for nonparametric regression, but it is incremental as it extends existing results to a specific discrete set without fundamentally altering the theoretical framework.
The paper tackles the problem of approximating smooth functions using deep neural networks with restricted parameter sets, showing that networks with parameters limited to {0, ±1/2, ±1, 2} can achieve similar approximation error and convergence rates as those with parameters in [-1,1], with depth, width, and active parameters scaling comparably up to logarithmic factors.
In this paper it is shown that $C_β$-smooth functions can be approximated by deep neural networks with ReLU activation function and with parameters $\{0,\pm \frac{1}{2}, \pm 1, 2\}$. The $l_0$ and $l_1$ parameter norms of considered networks are thus equivalent. The depth, width and the number of active parameters of the constructed networks have, up to a logarithmic factor, the same dependence on the approximation error as the networks with parameters in $[-1,1]$. In particular, this means that the nonparametric regression estimation with the constructed networks attains the same convergence rate as with sparse networks with parameters in $[-1,1]$.