Universal Acoustic Adversarial Attacks for Flexible Control of Speech-LLMs
This work exposes critical security flaws in speech LLMs, posing risks for applications relying on spoken language processing, and is incremental in extending adversarial attacks to selective control.
The paper tackles the vulnerability of speech large language models (LLMs) to universal acoustic adversarial attacks, where prepending a fixed audio segment can cause models like Qwen2-Audio and Granite-Speech to produce no output, override prompts, or selectively activate based on attributes like speaker gender or language.
The combination of pre-trained speech encoders with large language models has enabled the development of speech LLMs that can handle a wide range of spoken language processing tasks. While these models are powerful and flexible, this very flexibility may make them more vulnerable to adversarial attacks. To examine the extent of this problem, in this work we investigate universal acoustic adversarial attacks on speech LLMs. Here a fixed, universal, adversarial audio segment is prepended to the original input audio. We initially investigate attacks that cause the model to either produce no output or to perform a modified task overriding the original prompt. We then extend the nature of the attack to be selective so that it activates only when specific input attributes, such as a speaker gender or spoken language, are present. Inputs without the targeted attribute should be unaffected, allowing fine-grained control over the model outputs. Our findings reveal critical vulnerabilities in Qwen2-Audio and Granite-Speech and suggest that similar speech LLMs may be susceptible to universal adversarial attacks. This highlights the need for more robust training strategies and improved resistance to adversarial attacks.