Artificial Neural Networks generated by Low Discrepancy Sequences
This method reduces computational costs for neural network training, benefiting researchers and practitioners in machine learning, though it is incremental in optimizing existing sparse training approaches.
The paper tackles the problem of training sparse neural networks efficiently by generating them as deterministic paths using low discrepancy sequences, achieving accuracy comparable to dense networks with significantly lower computational complexity.
Artificial neural networks can be represented by paths. Generated as random walks on a dense network graph, we find that the resulting sparse networks allow for deterministic initialization and even weights with fixed sign. Such networks can be trained sparse from scratch, avoiding the expensive procedure of training a dense network and compressing it afterwards. Although sparse, weights are accessed as contiguous blocks of memory. In addition, enumerating the paths using deterministic low discrepancy sequences, for example the Sobol' sequence, amounts to connecting the layers of neural units by progressive permutations, which naturally avoids bank conflicts in parallel computer hardware. We demonstrate that the artificial neural networks generated by low discrepancy sequences can achieve an accuracy within reach of their dense counterparts at a much lower computational complexity.