Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference
This work addresses a critical blind spot in evaluating visual token pruning for MLLMs, specifically for OCR-critical applications, by introducing a spatial provenance audit that reveals quality-risk-traceability frontiers not captured by accuracy alone, which is important for developers and users relying on the interpretability and reliability of MLLM outputs.
This paper introduces an evidence-risk audit for visual token pruning in text-rich MLLMs, focusing on whether retained tokens are traceable to OCR regions supporting correct answers, rather than just answer accuracy. They found that while Qwen Target at 30% retention had similar accuracy to Full (0.786 vs 0.783), its positive-support coverage was significantly higher (0.620) compared to Random (0.270) and Grid (0.318) at the same budget, and achieved up to 4.32x batch-prefill speedup and 76.4% lower incremental peak memory.
Visual-token pruning is usually judged by answer quality at a fixed retention budget. For text-rich multimodal large language models (MLLMs), this protocol can miss a distinct failure: an answer remains correct even when no retained token is locally traceable to the small OCR region that supports it. We turn this blind spot into an evidence-risk audit that couples answer behavior with geometric token-origin provenance, interventions, and realized cost; transparent training-free selectors isolate controlled operating points. On locked image-disjoint confirmation, Qwen Target at 30% retention has observed accuracy 0.786 versus 0.783 for Full (paired image-cluster difference +0.003, 95% CI [-0.014, +0.020]), yet same-budget Target, Random, and Grid retain sharply different positive-support coverage: 0.620, 0.270, and 0.318. Across Qwen3-VL-8B, LLaVA-1.5-7B, and InternVL3.5-8B, matched controls, interventions, detector tests, and external methods reveal model-specific quality-risk-traceability frontiers that accuracy alone does not expose. Materialized prefixes yield up to 4.32x batch-prefill speedup and 76.4% lower incremental peak memory; full-validation TextVQA and DocVQA further show that favorable target-verification points do not imply task-general compression. Visual-token pruning should therefore report surviving spatial provenance and realized cost alongside quality and compression.