Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression
This work addresses the critical problem of deploying large multimodal models on resource-constrained edge devices, representing an incremental improvement in compression techniques.
The paper tackles the challenge of compressing large multimodal models for edge deployment by proposing an adaptive algorithm that optimizes sparsity and KV cache compression, achieving superior memory efficiency over state-of-the-art methods like SparseGPT and Wanda on benchmarks such as LLaVA-1.5 7B and 13B without compromising accuracy.
Large multimodal models (LMMs) have advanced significantly by integrating visual encoders with extensive language models, enabling robust reasoning capabilities. However, compressing LMMs for deployment on edge devices remains a critical challenge. In this work, we propose an adaptive search algorithm that optimizes sparsity and KV cache compression to enhance LMM efficiency. Utilizing the Tree-structured Parzen Estimator, our method dynamically adjusts pruning ratios and KV cache quantization bandwidth across different LMM layers, using model performance as the optimization objective. This approach uniquely combines pruning with key-value cache quantization and incorporates a fast pruning technique that eliminates the need for additional fine-tuning or weight adjustments, achieving efficient compression without compromising accuracy. Comprehensive evaluations on benchmark datasets, including LLaVA-1.5 7B and 13B, demonstrate our method superiority over state-of-the-art techniques such as SparseGPT and Wanda across various compression levels. Notably, our framework automatic allocation of KV cache compression resources sets a new standard in LMM optimization, delivering memory efficiency without sacrificing much performance.