Steal the Patch Size: Adversarially Manipulate Vision-Language Models
For security researchers and VLM deployers, this attack exposes a new vulnerability in model privacy, though it is incremental as it extends known side-channel techniques to a new domain.
This paper presents a black-box attack that recovers private vision-tokenizer configurations (patch size, preprocessing pipeline) of VLMs by exploiting a side channel from ViT patchification. The attack reliably infers these parameters across open-source and proprietary models, enabling downstream adversarial manipulation.
We present a black-box model-stealing attack that recovers private vision-tokenizer configurations of deployed vision-language models (VLMs), including the visual patch size and input preprocessing pipeline. The key idea is a task-level side channel induced by ViT-style patchification: when a synthetic grid image is aligned with the hidden patch grid, boundary cues are erased at tokenization, causing periodic accuracy drop. By sweeping the grid cell size and measuring these collapses, we infer the patch size; by introducing padding and a consistency-check test, we further identify whether preprocessing is dynamic- or fixed-resolution and recover the target resize resolution. Across open-source Qwen-VL variants and proprietary models including GPT and Claude, we reliably recover tokenizer-related parameters. Finally, we show that such leakage enables preprocessing-aware transfer attacks and model-targeted adversarial manipulation.