On the security relevance of weights in deep learning
This reveals a security vulnerability in deep learning that could affect practitioners relying on pre-trained or shared models, though it is incremental in expanding known attack vectors.
The paper demonstrates that a task-independent permutation of initial weights can severely degrade model accuracy (e.g., from over 90% to 50% on Fashion MNIST), showing this threat is broader than previously known weight-based attacks. The attack succeeds with high likelihood, is data-independent, and is hard to detect via standard metrics.
Recently, a weight-based attack on stochastic gradient descent inducing overfitting has been proposed. We show that the threat is broader: A task-independent permutation on the initial weights suffices to limit the achieved accuracy to for example 50% on the Fashion MNIST dataset from initially more than $90$%. These findings are confirmed on MNIST and CIFAR. We formally confirm that the attack succeeds with high likelihood and does not depend on the data. Empirically, weight statistics and loss appear unsuspicious, making it hard to detect the attack if the user is not aware. Our paper is thus a call for action to acknowledge the importance of the initial weights in deep learning.