Beyond the False Trade-off: Adaptive EWC for Stealthy and Generalizable T2I Backdoors
For researchers in adversarial machine learning, this work addresses the fidelity-ASR trade-off in backdoor attacks on T2I models, offering a method that improves both stealthiness and generalization.
The paper introduces Elastic Weight Consolidation (EWC) for stealthy backdoor attacks in text-to-image models, showing that standard static EWC creates a trade-off between attack success rate and fidelity. They propose Cosine-Aware Adaptive EWC, which dynamically adjusts regularization to maintain high ASR while preserving fidelity, achieving improved balance and robustness on out-of-domain datasets.
Preserving model fidelity is essential for stealthy text-to-image (T2I) backdoor attacks. Existing methods such as Learning without Forgetting (LwF) rely on output-based distillation, which provides limited regularization. We introduce Elastic Weight Consolidation (EWC) as a parameter-based alternative for preserving fidelity in backdoor learning. While stronger in principle, we show that standard static EWC with a fixed regularization weight lambda and mean-squared utility loss creates an artificial trade-off between attack success rate (ASR) and fidelity, particularly degrading performance on weak triggers. To address this, we propose Cosine-Aware Adaptive EWC, which dynamically adjusts EWC regularization using a cosine-based semantic utility and adaptive scheduling. This approach transforms EWC from a fixed penalty into a context-sensitive constraint, maintaining high ASR while preserving model fidelity. Experiments demonstrate improved ASR-fidelity balance and enhanced robustness on out-of-domain (OOD) datasets compared to existing baselines.