Testing RadiX-Nets: Advances in Viable Sparse Topologies
This work addresses a gap in evaluating sparse networks for scalable machine learning, but it is incremental as it focuses on testing rather than introducing new methods.
The paper tackles the challenge of testing RadiX-Nets, a type of sparse deep neural network, by developing a testing suite in TensorFlow to analyze their performance, topology, and training behavior, revealing issues like inconsistent training in some models.
The exponential growth of data has sparked computational demands on ML research and industry use. Sparsification of hyper-parametrized deep neural networks (DNNs) creates simpler representations of complex data. Past research has shown that some sparse networks achieve similar performance as dense ones, reducing runtime and storage. RadiX-Nets, a subgroup of sparse DNNs, maintain uniformity which counteracts their lack of neural connections. Generation, independent of a dense network, yields faster asymptotic training and removes the need for costly pruning. However, little work has been done on RadiX-Nets, making testing challenging. This paper presents a testing suite for RadiX-Nets in TensorFlow. We test RadiX-Net performance to streamline processing in scalable models, revealing relationships between network topology, initialization, and training behavior. We also encounter "strange models" that train inconsistently and to lower accuracy while models of similar sparsity train well.