Noise Optimization for Artificial Neural Networks
This work addresses the problem of improving the robustness of Artificial Neural Networks against adversarial attacks for researchers and practitioners in machine learning, offering an incremental improvement to existing noise-addition techniques.
This paper proposes a new technique to optimize the standard deviation of Gaussian noise added to each neuron in an Artificial Neural Network (ANN). The method computes pathwise stochastic gradient estimates for noise levels as a byproduct of backpropagation, allowing simultaneous optimization of noise and synaptic weights with minimal extra computational cost. Numerical experiments show significant performance improvement in robustness against black-box and white-box attacks on various ANN structures and computer vision datasets.
Adding noises to artificial neural network(ANN) has been shown to be able to improve robustness in previous work. In this work, we propose a new technique to compute the pathwise stochastic gradient estimate with respect to the standard deviation of the Gaussian noise added to each neuron of the ANN. By our proposed technique, the gradient estimate with respect to noise levels is a byproduct of the backpropagation algorithm for estimating gradient with respect to synaptic weights in ANN. Thus, the noise level for each neuron can be optimized simultaneously in the processing of training the synaptic weights at nearly no extra computational cost. In numerical experiments, our proposed method can achieve significant performance improvement on robustness of several popular ANN structures under both black box and white box attacks tested in various computer vision datasets.