Approximating Lipschitz continuous functions with GroupSort neural networks
This addresses the need for robust neural networks in adversarial attacks and Wasserstein GANs, but it is incremental as it builds on existing GroupSort networks with theoretical analysis.
The paper tackled the problem of approximating Lipschitz continuous functions using GroupSort neural networks with weight constraints, showing they can represent any Lipschitz continuous piecewise linear functions and providing upper bounds on depth and size, with synthetic experiments demonstrating efficiency compared to ReLU networks.
Recent advances in adversarial attacks and Wasserstein GANs have advocated for use of neural networks with restricted Lipschitz constants. Motivated by these observations, we study the recently introduced GroupSort neural networks, with constraints on the weights, and make a theoretical step towards a better understanding of their expressive power. We show in particular how these networks can represent any Lipschitz continuous piecewise linear functions. We also prove that they are well-suited for approximating Lipschitz continuous functions and exhibit upper bounds on both the depth and size. To conclude, the efficiency of GroupSort networks compared with more standard ReLU networks is illustrated in a set of synthetic experiments.