Analytical aspects of non-differentiable neural networks
This work addresses theoretical problems for researchers in deep learning, focusing on hardware-optimized networks, but it is incremental as it builds on existing methods for non-differentiable activations.
The paper tackles the expressivity and approximation challenges of non-differentiable neural networks, proving that quantized neural networks (QNNs) can approximate Lipschitz functions as well as differentiable ones and developing stochastic regularization techniques with quantitative estimates.
Research in computational deep learning has directed considerable efforts towards hardware-oriented optimisations for deep neural networks, via the simplification of the activation functions, or the quantization of both activations and weights. The resulting non-differentiability (or even discontinuity) of the networks poses some challenging problems, especially in connection with the learning process. In this paper, we address several questions regarding both the expressivity of quantized neural networks and approximation techniques for non-differentiable networks. First, we answer in the affirmative the question of whether QNNs have the same expressivity as DNNs in terms of approximation of Lipschitz functions in the $L^{\infty}$ norm. Then, considering a continuous but not necessarily differentiable network, we describe a layer-wise stochastic regularisation technique to produce differentiable approximations, and we show how this approach to regularisation provides elegant quantitative estimates. Finally, we consider networks defined by means of Heaviside-type activation functions, and prove for them a pointwise approximation result by means of smooth networks under suitable assumptions on the regularised activations.