Universal Lipschitz Approximation in Bounded Depth Neural Networks
This addresses the need for certified robust models in machine learning to mitigate adversarial attacks, representing a novel method rather than an incremental improvement.
The paper tackles the problem of adversarial attacks on machine learning models by proving that three-layer neural networks with FullSort activation functions are Universal Lipschitz function Approximators (ULAs), enabling provably robust models. Experimental results show ULAs are a competitive approach for certified classifiers.
Adversarial attacks against machine learning models are a rather hefty obstacle to our increasing reliance on these models. Due to this, provably robust (certified) machine learning models are a major topic of interest. Lipschitz continuous models present a promising approach to solving this problem. By leveraging the expressive power of a variant of neural networks which maintain low Lipschitz constants, we prove that three layer neural networks using the FullSort activation function are Universal Lipschitz function Approximators (ULAs). This both explains experimental results and paves the way for the creation of better certified models going forward. We conclude by presenting experimental results that suggest that ULAs are a not just a novelty, but a competitive approach to providing certified classifiers, using these results to motivate several potential topics of further research.