Defending Against Harmful Supervision Hidden in Benign Samples
For AI safety researchers, this work addresses a novel attack vector that bypasses current guardrails, though the proposed defense is incremental.
The paper identifies a vulnerability where harmful supervision can be hidden inside benign training samples, evading existing defenses. It proposes DR-SFT, a method that uses token-level regularization to mitigate harmful fine-tuning, demonstrating improved robustness over coarse filtering.
Existing defenses are effective when harmful content is explicitly mixed into downstream fine-tuning data, but crafted samples can instead hide harmful supervision inside benign tasks. We propose Embedded Attack, where harmful QA pairs are embedded within benign training samples, and show that representative guardrails often fail to detect them at the example level. To address this, we propose Dual-Reference SFT (DR-SFT), which adapts DPO-style contrastive objective design to SFT through token-level regularization, mitigating harmful fine-tuning beyond coarse data filtering.