Nerva: a Truly Sparse Implementation of Neural Networks
This work addresses computational efficiency for machine learning practitioners using sparse neural networks, though it appears incremental as it builds on existing sparse matrix libraries.
The authors tackled the problem of inefficient sparse neural network training by introducing Nerva, a C++ library that uses Intel MKL sparse matrix operations instead of binary masks. They demonstrated that Nerva reduces training time by 4x and memory usage while maintaining equivalent accuracy to PyTorch on CIFAR-10 at 99% sparsity.
We introduce Nerva, a fast neural network library under development in C++. It supports sparsity by using the sparse matrix operations of Intel's Math Kernel Library (MKL), which eliminates the need for binary masks. We show that Nerva significantly decreases training time and memory usage while reaching equivalent accuracy to PyTorch. We run static sparse experiments with an MLP on CIFAR-10. On high sparsity levels like $99\%$, the runtime is reduced by a factor of $4\times$ compared to a PyTorch model using masks. Similar to other popular frameworks such as PyTorch and Keras, Nerva offers a Python interface for users to work with.