R3Net: Random Weights, Rectifier Linear Units and Robustness for Artificial Neural Network
This work addresses robustness to input perturbations for neural network users, but it appears incremental as it builds on existing randomized and ReLU-based methods.
The authors tackled the problem of ensuring robustness and discriminability in neural networks by proposing R3Net, an architecture with random weights and ReLU activations, and proved that it exhibits Lipschitz continuity and maintains distance between outputs for different inputs.
We consider a neural network architecture with randomized features, a sign-splitter, followed by rectified linear units (ReLU). We prove that our architecture exhibits robustness to the input perturbation: the output feature of the neural network exhibits a Lipschitz continuity in terms of the input perturbation. We further show that the network output exhibits a discrimination ability that inputs that are not arbitrarily close generate output vectors which maintain distance between each other obeying a certain lower bound. This ensures that two different inputs remain discriminable while contracting the distance in the output feature space.