Shape and Substance: Dual-Layer Side-Channel Attacks on Local Vision-Language Models
This work addresses a critical security problem for users of local VLMs, revealing a novel algorithmic side-channel that compromises data privacy despite on-device execution, with incremental implications for Edge AI security design.
The paper tackles the vulnerability of on-device Vision-Language Models (VLMs) to side-channel attacks by exploiting dynamic preprocessing, demonstrating that attackers can infer input geometry and semantic content using execution-time variations and cache contention, achieving reliable inference of privacy-sensitive contexts in models like LLaVA-NeXT and Qwen2-VL.
On-device Vision-Language Models (VLMs) promise data privacy via local execution. However, we show that the architectural shift toward Dynamic High-Resolution preprocessing (e.g., AnyRes) introduces an inherent algorithmic side-channel. Unlike static models, dynamic preprocessing decomposes images into a variable number of patches based on their aspect ratio, creating workload-dependent inputs. We demonstrate a dual-layer attack framework against local VLMs. In Tier 1, an unprivileged attacker can exploit significant execution-time variations using standard unprivileged OS metrics to reliably fingerprint the input's geometry. In Tier 2, by profiling Last-Level Cache (LLC) contention, the attacker can resolve semantic ambiguity within identical geometries, distinguishing between visually dense (e.g., medical X-rays) and sparse (e.g., text documents) content. By evaluating state-of-the-art models such as LLaVA-NeXT and Qwen2-VL, we show that combining these signals enables reliable inference of privacy-sensitive contexts. Finally, we analyze the security engineering trade-offs of mitigating this vulnerability, reveal substantial performance overhead with constant-work padding, and propose practical design recommendations for secure Edge AI deployments.