LOP: Learning Optimal Pruning for Efficient On-Demand MLLMs Scaling
This addresses the need for efficient on-demand adaptation of MLLMs across hardware platforms, representing a novel method for a known bottleneck rather than a foundational advancement.
The paper tackles the problem of high computational overhead in structural pruning for multimodal large language models (MLLMs) by proposing LOP, a framework that learns optimal pruning strategies directly from constraints, eliminating iterative searches and achieving up to three orders of magnitude speedup while outperforming state-of-the-art methods.
Structural pruning techniques are essential for deploying multimodal large language models (MLLMs) across various hardware platforms, from edge devices to cloud servers. However, current pruning methods typically determine optimal strategies through iterative search processes, resulting in substantial computational overhead for on-demand MLLMs adaptation. To address this challenge, we propose LOP, an efficient neural pruning framework that learns optimal pruning strategies from the target pruning constraint, eliminating the need for computationally expensive search-based methods. LOP approach trains autoregressive neural networks (NNs) to directly predict layer-wise pruning strategies adaptive to the target pruning constraint, eliminating the time-consuming iterative searches. Experimental results across multiple tasks show that LOP outperforms state-of-the-art pruning methods in various metrics while achieving up to three orders of magnitude speedup.