Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
This work addresses efficient deployment of VLMs for resource-constrained applications, representing a strong incremental improvement over existing quantization methods.
The paper tackles the problem of costly deployment of Vision-Language Models (VLMs) by proposing GRACE, a quantization-aware training framework that unifies knowledge distillation and quantization under the Information Bottleneck principle, achieving INT4 models that outperform FP16 baselines (e.g., 70.1 vs. 66.8 on SQA for LLaVA-1.5-7B) with 3× throughput and 54% memory reduction.
Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss. Despite its potential, quantization-aware training for VLMs remains underexplored. We propose GRACE, a framework unifying knowledge distillation and QAT under the Information Bottleneck principle: quantization constrains information capacity while distillation guides what to preserve within this budget. Treating the teacher as a proxy for task-relevant information, we introduce confidence-gated decoupled distillation to filter unreliable supervision, relational centered kernel alignment to transfer visual token structures, and an adaptive controller via Lagrangian relaxation to balance fidelity against capacity constraints. Across extensive benchmarks on LLaVA and Qwen families, our INT4 models consistently outperform FP16 baselines (e.g., LLaVA-1.5-7B: 70.1 vs. 66.8 on SQA; Qwen2-VL-2B: 76.9 vs. 72.6 on MMBench), nearly matching teacher performance. Using real INT4 kernel, we achieve 3$\times$ throughput with 54% memory reduction. This principled framework significantly outperforms existing quantization methods, making GRACE a compelling solution for resource-constrained deployment.