Improving compute efficacy frontiers with SliceOut
This addresses compute bottlenecks for tasks requiring frequent model re-training or large-scale training workloads, offering incremental improvements in efficiency.
The paper tackles the problem of improving compute efficiency in deep learning training by introducing SliceOut, a dropout-inspired method that drops contiguous sets of units to leverage GPU memory layout. It achieves 10-40% speedups and memory reduction across models like Wide ResNets and Transformers with minimal accuracy loss.
Pushing forward the compute efficacy frontier in deep learning is critical for tasks that require frequent model re-training or workloads that entail training a large number of models. We introduce SliceOut -- a dropout-inspired scheme designed to take advantage of GPU memory layout to train deep learning models faster without impacting final test accuracy. By dropping contiguous sets of units at random, our method realises training speedups through (1) fast memory access and matrix multiplication of smaller tensors, and (2) memory savings by avoiding allocating memory to zero units in weight gradients and activations. At test time, turning off SliceOut performs an implicit ensembling across a linear number of architectures that preserves test accuracy. We demonstrate 10-40% speedups and memory reduction with Wide ResNets, EfficientNets, and Transformer models, with minimal to no loss in accuracy. This leads to faster processing of large computational workloads overall, and significantly reduce the resulting energy consumption and CO2emissions.