Unlearning the Unpromptable: Prompt-free Instance Unlearning in Diffusion Models
This addresses a practical need for diffusion model providers to ensure privacy protection and ethical compliance by enabling selective forgetting of unpromptable outputs, representing an incremental advance in machine unlearning.
The paper tackles the problem of instance unlearning for undesired but unpromptable outputs in diffusion models, such as specific faces or culturally inaccurate depictions, and introduces a prompt-free method that effectively forgets these outputs while preserving model integrity, as demonstrated on models like Stable Diffusion 3 and DDPM-CelebA.
Machine unlearning aims to remove specific outputs from trained models, often at the concept level, such as forgetting all occurrences of a particular celebrity or filtering content via text prompts. However, many undesired outputs, such as an individual's face or generations culturally or factually misinterpreted, cannot often be specified by text prompts. We address this underexplored setting of instance unlearning for outputs that are undesired but unpromptable, where the goal is to forget target outputs selectively while preserving the rest. To this end, we introduce an effective surrogate-based unlearning method that leverages image editing, timestep-aware weighting, and gradient surgery to guide trained diffusion models toward forgetting specific outputs. Experiments on conditional (Stable Diffusion 3) and unconditional (DDPM-CelebA) diffusion models demonstrate that our prompt-free method uniquely unlearns unpromptable outputs, such as faces and culturally inaccurate depictions, with preserved integrity, unlike prompt-based and prompt-free baselines. Our proposed method would serve as a practical hotfix for diffusion model providers to ensure privacy protection and ethical compliance.