Dual-side Sparse Tensor Core
This addresses the challenge of accelerating DNN inference by exploiting dual-side sparsity, offering a novel architectural solution for hardware designers and AI practitioners, though it builds incrementally on existing tensor core technology.
The paper tackled the problem of efficiently leveraging both weight and activation sparsity in deep neural network inference, which existing GPUs cannot handle for dynamic activations, and resulted in a design that improves performance by up to one order of magnitude with minimal hardware overhead.
Leveraging sparsity in deep neural network (DNN) models is promising for accelerating model inference. Yet existing GPUs can only leverage the sparsity from weights but not activations, which are dynamic, unpredictable, and hence challenging to exploit. In this work, we propose a novel architecture to efficiently harness the dual-side sparsity (i.e., weight and activation sparsity). We take a systematic approach to understand the (dis)advantages of previous sparsity-related architectures and propose a novel, unexplored paradigm that combines outer-product computation primitive and bitmap-based encoding format. We demonstrate the feasibility of our design with minimal changes to the existing production-scale inner-product-based Tensor Core. We propose a set of novel ISA extensions and co-design the matrix-matrix multiplication and convolution algorithms, which are the two dominant computation patterns in today's DNN models, to exploit our new dual-side sparse Tensor Core. Our evaluation shows that our design can fully unleash the dual-side DNN sparsity and improve the performance by up to one order of magnitude with \hl{small} hardware overhead.