BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference
This addresses efficiency problems for deploying large AI models in resource-constrained environments, though it is incremental as it builds on existing structured matrix methods.
The paper tackles the computational challenges of dense matrix-vector operations in large foundation models during inference by introducing the Block-Level Adaptive Structured (BLAST) matrix, which compresses models with results like 70% complexity reduction for ViT and 2x compression for Llama-7B with minimal performance degradation.
Large-scale foundation models have demonstrated exceptional performance in language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, we introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures prevalent in the weight matrices of linear layers within deep learning models. Compared to existing structured matrices, the BLAST matrix offers substantial flexibility, as it can represent various types of structures that are either learned from data or computed from pre-existing weight matrices. We demonstrate the efficiency of using the BLAST matrix for compressing both language and vision tasks, showing that (i) for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively; and (ii) for large foundation models such as Llama-7B and DiT-XL, the BLAST matrix achieves a 2x compression while exhibiting the lowest performance degradation among all tested structured matrices. Our code is available at https://github.com/changwoolee/BLAST.