No-Skim: Towards Efficiency Robustness Evaluation on Skimming-based Language Models
This work addresses a security and efficiency problem for users of skimming-based LLMs, revealing a novel vulnerability that could impact deployment costs and robustness.
The paper tackles the vulnerability of skimming-based language models to Denial-of-Service attacks by proposing No-Skim, a framework that generates adversarial inputs to increase computation costs, resulting in an average 145% increase in running cost in worst-case scenarios.
To reduce the computation cost and the energy consumption in large language models (LLM), skimming-based acceleration dynamically drops unimportant tokens of the input sequence progressively along layers of the LLM while preserving the tokens of semantic importance. However, our work for the first time reveals the acceleration may be vulnerable to Denial-of-Service (DoS) attacks. In this paper, we propose No-Skim, a general framework to help the owners of skimming-based LLM to understand and measure the robustness of their acceleration scheme. Specifically, our framework searches minimal and unnoticeable perturbations at character-level and token-level to generate adversarial inputs that sufficiently increase the remaining token ratio, thus increasing the computation cost and energy consumption. We systematically evaluate the vulnerability of the skimming acceleration in various LLM architectures including BERT and RoBERTa on the GLUE benchmark. In the worst case, the perturbation found by No-Skim substantially increases the running cost of LLM by over 145% on average. Moreover, No-Skim extends the evaluation framework to various scenarios, making the evaluation conductible with different level of knowledge.