Gradient-Free Structured Pruning with Unlabeled Data
This addresses the need for efficient inference in customized LLMs, offering a faster alternative to methods requiring labeled data or retraining, though it is incremental as it builds on existing pruning techniques.
The paper tackles the problem of high computation cost and inference latency in Large Language Models by proposing a gradient-free structured pruning framework that uses only unlabeled data, achieving up to 40% reduction in FLOP count with less than 4% accuracy loss on GLUE and SQuAD benchmarks.
Large Language Models (LLMs) have achieved great success in solving difficult tasks across many domains, but such success comes with a high computation cost, and inference latency. As developers and third parties customize these models, the need to provide efficient inference has increased. Many efforts have attempted to reduce inference cost through model compression techniques such as pruning and distillation. However, these techniques either require labeled data, or are time-consuming as they require the compressed model to be retrained to regain accuracy. In this paper, we propose a gradient-free structured pruning framework that uses only unlabeled data. An evaluation on the GLUE and SQuAD benchmarks using BERT$_{BASE}$ and DistilBERT illustrates the effectiveness of the proposed approach. By only using the weights of the pre-trained model and unlabeled data, in a matter of a few minutes on a single GPU, up to 40% of the original FLOP count can be reduced with less than a 4% accuracy loss across all tasks considered.