HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models
This work is significant for researchers and practitioners deploying efficient VLMs, as it offers a method to mitigate object hallucinations amplified by compression, which is a critical VLM-specific failure mode.
This paper addresses the challenge of pruning vision-language models (VLMs) by developing HiPP-Prune, a hierarchical preference-conditioned structured pruning framework. It enables queryable trade-offs between task utility and hallucination robustness, demonstrating controllable robustness-utility trade-offs on LLaVA with POPE and ScienceQA under matched sparsity budgets.
Pruning vision-language models (VLMs) for efficient deployment is challenging because compression can affect not only task utility but also visual grounding, often amplifying object hallucinations even at the same sparsity level. We present HiPP-Prune, a hierarchical preference-conditioned structured pruning framework that treats pruning as conditional resource allocation under multiple objectives. HiPP-Prune makes plan-level decisions: a single policy invocation outputs a global pruning blueprint by factorizing decisions into an overall sparsity budget and a layer-wise allocation, enabling queryable trade-offs via a user-specified preference vector. To account for VLM-specific failure modes, our policy state integrates a visual sensitivity signal derived from attention flow between vision tokens and language hidden states, discouraging over-pruning of vision-critical layers that facilitate cross-modal fusion. We optimize pruning plans with plan-level Group Relative Policy Optimization (GRPO) under a multi-objective return that combines task utility, hallucination robustness (POPE), compression, and a synaptic-flow-inspired stability proxy to reduce unproductive exploration in high-sparsity regimes. Experiments on LLaVA with POPE and ScienceQA demonstrate that HiPP-Prune discovers diverse non-dominated pruning plans and provides controllable robustness--utility trade-offs under matched sparsity budgets.