Approximation theory for 1-Lipschitz ResNets
This provides a rigorous foundation for using 1-Lipschitz ResNets in generative modeling, inverse problems, and robust classification, though it is incremental in establishing theoretical guarantees.
The paper tackles the problem of approximating 1-Lipschitz functions using residual networks, proving that 1-Lipschitz ResNets are dense in the set of such functions on compact domains and can exactly represent piecewise affine ones, with results extended to fixed-width networks by inserting norm-constrained linear maps.
1-Lipschitz neural networks are fundamental for generative modelling, inverse problems, and robust classifiers. In this paper, we focus on 1-Lipschitz residual networks (ResNets) based on explicit Euler steps of negative gradient flows and study their approximation capabilities. Leveraging the Restricted Stone-Weierstrass Theorem, we first show that these 1-Lipschitz ResNets are dense in the set of scalar 1-Lipschitz functions on any compact domain when width and depth are allowed to grow. We also show that these networks can exactly represent scalar piecewise affine 1-Lipschitz functions. We then prove a stronger statement: by inserting norm-constrained linear maps between the residual blocks, the same density holds when the hidden width is fixed. Because every layer obeys simple norm constraints, the resulting models can be trained with off-the-shelf optimisers. This paper provides the first universal approximation guarantees for 1-Lipschitz ResNets, laying a rigorous foundation for their practical use.