dMath: Distributed Linear Algebra for DL
This work addresses the need for efficient and scalable deep learning tools for researchers and practitioners, though it is incremental as it builds on existing distributed computing concepts.
The paper tackles the challenge of scaling deep learning computations by introducing dMath, a distributed linear algebra library that achieves leading scaling performance using intranode, internode, and hybrid parallelism, with persistent GPU memory storage and advanced management to reduce costly data transfers.
The paper presents a parallel math library, dMath, that demonstrates leading scaling when using intranode, internode, and hybrid-parallelism for deep learning (DL). dMath provides easy-to-use distributed primitives and a variety of domain-specific algorithms including matrix multiplication, convolutions, and others allowing for rapid development of scalable applications like deep neural networks (DNNs). Persistent data stored in GPU memory and advanced memory management techniques avoid costly transfers between host and device. dMath delivers performance, portability, and productivity to its specific domain of support.