The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit
This addresses fault tolerance in neural networks, particularly relevant for neuromorphic hardware and biological networks, but the approach is incremental as it builds on existing probabilistic methods.
The paper tackles the problem of quantifying how many random neuron failures a neural network can tolerate before output error becomes significant, deriving a provable bound on output error under small random Bernoulli crashes using Taylor expansion in the continuous limit. It shows that the theoretical bounds can compare fault tolerance across architectures and design a regularizer to improve it, with experimental validation provided.
The loss of a few neurons in a brain rarely results in any visible loss of function. However, the insight into what "few" means in this context is unclear. How many random neuron failures will it take to lead to a visible loss of function? In this paper, we address the fundamental question of the impact of the crash of a random subset of neurons on the overall computation of a neural network and the error in the output it produces. We study fault tolerance of neural networks subject to small random neuron/weight crash failures in a probabilistic setting. We give provable guarantees on the robustness of the network to these crashes. Our main contribution is a bound on the error in the output of a network under small random Bernoulli crashes proved by using a Taylor expansion in the continuous limit, where close-by neurons at a layer are similar. The failure mode we adopt in our model is characteristic of neuromorphic hardware, a promising technology to speed up artificial neural networks, as well as of biological networks. We show that our theoretical bounds can be used to compare the fault tolerance of different architectures and to design a regularizer improving the fault tolerance of a given architecture. We design an algorithm achieving fault tolerance using a reasonable number of neurons. In addition to the theoretical proof, we also provide experimental validation of our results and suggest a connection to the generalization capacity problem.