Backdoor Attacks Against Speech Language Models
This addresses a security problem for users of multimodal AI systems, particularly in speech processing, and is incremental as it extends backdoor attack studies to audio domains.
The paper tackles the vulnerability of speech language models to audio backdoor attacks, demonstrating high success rates from 90.76% to 99.41% across multiple encoders, datasets, and tasks, and proposes a fine-tuning-based defense.
Large Language Models (LLMs) and their multimodal extensions are becoming increasingly popular. One common approach to enable multimodality is to cascade domain-specific encoders with an LLM, making the resulting model inherit vulnerabilities from all of its components. In this work, we present the first systematic study of audio backdoor attacks against speech language models. We demonstrate its effectiveness across four speech encoders and three datasets, covering four tasks: automatic speech recognition (ASR), speech emotion recognition, and gender and age prediction. The attack consistently achieves high success rates, ranging from 90.76% to 99.41%. To better understand how backdoors propagate, we conduct a component-wise analysis to identify the most vulnerable stages of the pipeline. Finally, we propose a fine-tuning-based defense that mitigates the threat of poisoned pretrained encoders.