Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors
For practitioners reusing CLIP models, this work reveals that native backdoor success does not guarantee risk across interfaces, and provides a method to create backdoors that specifically target text-conditioned reuse.
Existing CLIP backdoor attacks are only validated on a single native task, leaving their exposure across diverse deployment interfaces (e.g., feature extraction, retrieval) unknown. The paper introduces DIFE, an auditing framework that reveals a structured landscape of backdoor exposure, and proposes BadTextTower, a backdoor that achieves strong text-conditioned retrieval/reranking/selection exposure while keeping visual-only reuse clean.
Contrastive Language-Image Pre-training models are widely reused across downstream interfaces, including feature extraction, retrieval, reranking, and selection. Existing CLIP backdoor, however, usually validate attacks on a small attack-native task, leaving unclear whether the same poisoned checkpoint remains exposed, weakens, or becomes not applicable when reused through other interfaces. We introduce DIFE, a Deployment-Interface Footprint Evaluation framework that audits backdoored CLIP checkpoints across deployment interfaces. DIFE makes various evaluations comparable by specifying each interface's component readout, trigger channel, target event, reference condition, and metric. DIFE also introduces effective-footprint diagnosis to identify the reusable CLIP component or component combination that carries exposure and explains where risk transfers. Auditing reproduced CLIP backdoors with DIFE reveals a structured landscape: native success is not a checkpoint-level risk certificate, exposure follows component footprints, text-side poisoning does not yield textual-encoder control, and some coupled attacks remain mechanism-bound. This audit reveals a import gapin existing CLIP backdoors: a textual encoder that itself becomes a reusable carrier of adversarial behavior. We therefore introduce BadTextTower to fill this gap. BadTextTower produces strong text-conditioned retrieval, reranking, and selection exposure while leaving visual-only reuse nearly clean.