Effective Approaches to Batch Parallelization for Dynamic Neural Network Architectures
This work addresses a computational bottleneck for researchers and practitioners using dynamic architectures, offering significant performance improvements, though it is incremental in building on existing batching methods.
The paper tackles the challenge of efficiently batching dynamic neural network architectures, achieving speedups of over 10x consistently and up to 1000x in specific cases like sparsely gated mixture of experts layers.
We present a simple dynamic batching approach applicable to a large class of dynamic architectures that consistently yields speedups of over 10x. We provide performance bounds when the architecture is not known a priori and a stronger bound in the special case where the architecture is a predetermined balanced tree. We evaluate our approach on Johnson et al.'s recent visual question answering (VQA) result of his CLEVR dataset by Inferring and Executing Programs (IEP). We also evaluate on sparsely gated mixture of experts layers and achieve speedups of up to 1000x over the naive implementation.