On the Permanence of Backdoors in Evolving Models
This addresses the problem of backdoor persistence in practical, evolving models for security researchers, showing that existing static assumptions are incomplete.
This paper investigates how backdoor attacks behave in continuously evolving neural networks that are fine-tuned to adapt to data drifts, finding that fine-tuning progressively erases injected backdoors and demonstrating that novel fine-tuning strategies can accelerate this forgetting process.
Existing research on training-time attacks for deep neural networks (DNNs), such as backdoors, largely assume that models are static once trained, and hidden backdoors trained into models remain active indefinitely. In practice, models are rarely static but evolve continuously to address distribution drifts in the underlying data. This paper explores the behavior of backdoor attacks in time-varying models, whose model weights are continually updated via fine-tuning to adapt to data drifts. Our theoretical analysis shows how fine-tuning with fresh data progressively "erases" the injected backdoors, and our empirical study illustrates how quickly a time-varying model "forgets" backdoors under a variety of training and attack settings. We also show that novel fine-tuning strategies using smart learning rates can significantly accelerate backdoor forgetting. Finally, we discuss the need for new backdoor defenses that target time-varying models specifically.