D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
This addresses efficiency problems for MLLM users by improving token pruning for fine-grained tasks, though it appears incremental as it builds on existing pruning strategies.
The paper tackles the computational burden of processing long visual token sequences in Multimodal Large Language Models (MLLMs) by introducing D2Pruner, a token pruning framework that combines debiased importance with structural diversity. Results show it reduces FLOPs by 74.2% while retaining 99.2% performance on general tasks and maintains 85.7% performance at a 90% token reduction rate on localization benchmarks, with up to 63.53% improvement over existing methods.
Processing long visual token sequences poses a significant computational burden on Multimodal Large Language Models (MLLMs). While token pruning offers a path to acceleration, we find that current methods, while adequate for general understanding, catastrophically fail on fine-grained localization tasks. We attribute this failure to the inherent flaws of the two prevailing strategies: importance-based methods suffer from a strong positional bias, an inherent model artifact that distracts from semantic content, while diversity-based methods exhibit structural blindness, disregarding the user's prompt and spatial redundancy. To address this, we introduce D2Pruner, a framework that rectifies these issues by uniquely combining debiased importance with a structural pruning mechanism. Our method first secures a core set of the most critical tokens as pivots based on a debiased attention score. It then performs a Maximal Independent Set (MIS) selection on the remaining tokens, which are modeled on a hybrid graph where edges signify spatial proximity and semantic similarity. This process iteratively preserves the most important and available token while removing its neighbors, ensuring that the supplementary tokens are chosen to maximize importance and diversity. Extensive experiments demonstrate that D2Pruner has exceptional efficiency and fidelity. Applied to LLaVA-1.5-7B for general understanding tasks, it reduces FLOPs by 74.2\% while retaining 99.2\% of its original performance. Furthermore, in challenging localization benchmarks with InternVL-2.5-8B, it maintains 85.7\% performance at a 90\% token reduction rate, marking a significant advancement with up to 63. 53\% improvement over existing methods.