Efficient Sparse Training with Structured Dropout
This work addresses the need for efficient training methods in deep learning by offering a drop-in replacement for dropout that can speed up training on GPUs, though it is incremental as it builds on existing dropout techniques.
The paper tackles the problem of dropout's unstructured sparsity not translating to GPU speed-ups by introducing SparseDrop, a structured dropout variant, and shows it achieves faster training speeds with similar or better regularization, providing a CUDA implementation that yields speed-ups even at low sparsity levels.
Dropout is a common regularisation technique in deep learning that improves generalisation. Even though it introduces sparsity and thus potential for higher throughput, it usually cannot bring speed-ups on GPUs due to its unstructured nature. In this project, I experiment with SparseDrop, a structured, hardware-friendly variant of dropout that can exploit such sparsity. I provide a CUDA implementation of SparseDrop, achieving speed-ups against its dense counterpart even at low sparsity levels. The empirical results demonstrate that SparseDrop provides similar, or sometimes even better, regularisation properties as standard dropout. This suggests its potential as a drop-in replacement to standard dropout with faster training speeds. The source code is available at https://github.com/andylolu2/sparse-dropout